Jekyll2023-09-18T01:19:26+00:00https://www.akshaydeo.com/feed.xmlAkshay Deo👋 I am Akshay Deo, a computer engineering from Pune. Co-founded 3 Startups, built 20+ Products and currently working on Postman as Head of Product Engineering.Akshay DeoBuilding Developer Tools is Hard2023-04-02T08:00:46+00:002023-04-02T08:00:46+00:00https://www.akshaydeo.com/blog/2023/04/02/building-developer-tools-is-hard<blockquote>
<p>While anyone can certainly benefit from this post, its primary focus is on early-stage founders of developer tooling companies.</p>
</blockquote>
<p>Selling developer tools to developers is like selling paintbrushes to artists. Just as an artist needs quality tools to create their masterpiece, a developer needs quality tools to build their software. Both require specialized tools that are designed to enhance their craft and help them achieve their goals. Just as an artist needs to choose the right brush for a specific stroke, a developer needs to choose the right tool for a specific task. And just as an artist’s brush can make the difference between a good painting and a great one, a developer’s tool can make the difference between functional software and exceptional software.</p>
<p>It’s common for developers to overestimate their abilities and underestimate the complexity of the challenges they face. For example, Abhinav Asthana, the founder of Postman, once remarked that many people believed they could create Postman in a single weekend. However, what they typically have in mind is a simplified version of Postman based on their understanding. However, as they delve deeper into the development process, they discover that the reality is far more intricate than they had initially envisioned, and unexpected complexities inevitably arise.</p>
<p>As for any other company, your top priority is communicating your value proposition in a way that resonates with your target audience. While developers may be impressed by your product’s technical prowess, highlighting its tangible benefits and how it can significantly improve their workflow is essential.</p>
<p>With that in mind, there’s a particular micro-framework that I refer to as a litmus test for developer tools. You can employ this framework to inform various aspects of your product, such as packaging, sales pitch, website design, and product onboarding, among others.</p>
<hr />
<h2 id="the-number-of-touchpoints-required-for-adoptionevaluation-should-be-as-low-as-possible-ideally-one">The number of touchpoints required for adoption/evaluation should be as low as possible, ideally one</h2>
<hr />
<p>If you are building a tool similar to NewRelic, an infrastructure monitoring tool widely used by companies, the primary stakeholders for selling your product are likely the engineering teams and SREs. These individuals will benefit most directly from your tool’s value proposition. Additionally, the security team will play a critical role in evaluating the tool’s security features and ensuring compliance with relevant laws regarding data tenancy. Integrating or evaluating a tool like this can be time-consuming and costly, involving DevOps and engineering work. This makes the evaluation process longer and more difficult for both sides.</p>
<p>On the other hand, anyone from any team can adopt VSCode Editor editor without breaking anyone else’s workflow. Once they find it useful, they start advocating it within the team, and that’s how bottom-up adoption starts. Hence it’s easier for your customer to evaluate something like VSCode.</p>
<p>Understanding these touchpoints and personas can help guide your growth strategy. For example, if your product requires a high degree of customization and support, you may need a sales-driven approach. In contrast, if your product is easy to use and requires little setup, a self-serve/product-led approach may be more appropriate.</p>
<p>Identifying the target customer persona through these touchpoints is also essential, as different individuals within an organization may have different priorities and concerns. For example, a CTO may prioritize security and scalability, while a Head of Product may prioritize ease of use and feature sets. Knowing your target customer persona can help you tailor your product and messaging to meet their specific needs.</p>
<p>Prioritizing product development efforts is another critical aspect of understanding touchpoints and personas. For example, suppose your target customer persona is a security head; in that case, prioritizing security certifications and integrations with existing tooling ecosystems may be more important than adding new features.</p>
<p>Understanding touchpoints and personas can inform your pricing strategy. For example, if your target customer persona is engineers, offering a freemium model that allows them to try your product before committing to a paid plan may be more effective than requiring payment upfront.</p>
<p>By identifying and understanding the various touchpoints and personas involved in product adoption, you can optimize your growth strategy, target the right customers, prioritize development efforts, and design a pricing strategy that appeals to your target audience.</p>
<hr />
<h2 id="another-essential-aspect-is-how-well-your-product-integrates-with-the-existing-ecosystem-if-your-product-expects-too-many-behavior-adoptions-ie-higher-time-to-adopt-there-is-a-high-chance-of-failure-this-includes-time-to-value-returns-as-well">Another essential aspect is how well your product integrates with the existing ecosystem. If your product expects too many behavior adoptions (i.e. higher time-to-adopt), there is a high chance of failure. This includes time-to-value returns as well.</h2>
<hr />
<p><img class="img-responsive" src="/public/images/adoption_circle.png" width="100%" /></p>
<p>Let’s take the example of “co-pilot” for VSCode. The time required to try it out is almost negligible, just by installing the extension. The time to value is also immediate since it starts working as soon as you start typing. Moreover, you don’t need to alter your current ecosystem unless you are using a different editor.</p>
<p>However, if a user is not using VSCode, the journey to use “co-pilot” becomes a bit complex.</p>
<p>They would first need to download and set up VSCode and install any relevant extensions, which can be time-consuming and can result in a higher drop-off rate. Only after completing these steps can they join the same journey.</p>
<p>It’s not necessarily a problem, as every tool will have early adopters willing to put in a significant effort to adopt it. However, if the time to value is high, along with a lengthy adoption process, there is a greater risk of slow or no growth.</p>
<p>Developing developer tools is a challenging task. The rules that apply to other industries or sectors may not always translate well into the world of developer tools. However, these unique challenges make it an exciting and intriguing space.</p>
<blockquote>
<p>As a part of Sifar Ventures, I advise and invest in early-stage companies. Feel free to send me your ideas/products at akshay@akshaydeo.com.</p>
</blockquote>Akshay DeoWhile anyone can certainly benefit from this post, its primary focus is on early-stage founders of developer tooling companies.Excited to announce our first angel investment 🙌2021-11-05T08:00:46+00:002021-11-05T08:00:46+00:00https://www.akshaydeo.com/blog/2021/11/05/angel-investing-peerlist<h2 id="why-angel-investing">Why Angel Investing?</h2>
<ul>
<li>Is it because everyone around you is one?</li>
<li>Is it because you have money to invest?</li>
</ul>
<p>This is a fair question to ask. And I am constantly perplexed with my attempt to answer this question effectively. Of course, there is no correct answer. But one thing that I love the most is, solving problems. And in my lifetime, I can solve only a limited number of problems, and this is my attempt to maximize that number 🙂.</p>
<h2 id="ideology">Ideology</h2>
<p>Traditionally angel investing is looked at as a gamble where if the bet works out, one could get returns 50 - 1000x. So that’s one way to look at it. But with <a href="https://sifar.vc">Sifar Ventures</a>, my goal is to focus more on intangible gains one gets while working with early-stage startups.</p>
<p>I have been a founder in the past and understand the chaos inside and around the founders. What they look for is de-risking as many things as possible to find the Problem Market Fit and Product Market Fit. When I completed the journey with a failed startup, I realized that I had learned a lot during the process. Not just about the business but the mindset to go through this one-of-a-kind journey in an entrepreneur’s career and life.</p>
<p>As I have worked extensively on all the layers of technology and products, I realized that I could successfully de-risk the technical and product problems of an early-stage startup. I have taken multiple attempts at helping startups for free, for minimal equity, but none of them worked out as expected. <strong>As they say, giving something for free is the most expensive thing.</strong></p>
<h2 id="hence-sifar-ventures">Hence <a href="https://sifar.vc">Sifar Ventures</a></h2>
<p><a href="https://peerlist.io/vishwesh">@Vishwesh Jirgale</a> and I have worked together for 6-7 years. One common theme of our conversations is what problems we face daily and how we could solve them effectively 💡. We love tech, startups and we love entrepreneurs. A year ago, while brainstorming on a few ideas, We realized that we have a distinct set of capabilities to help early-stage startups de-risk technology, marketing, hiring, and fundraising. Hence last year, we decided to start Sifar Ventures, an early-stage VC firm focussing on (but not limited to) Indian Startups.</p>
<h2 id="we-are-proud-to-announce-our-first-investment-peerlist">We are proud to announce our first investment: <a href="https://peerlist.io?ref=amd">Peerlist</a></h2>
<p><img src="https://user-images.githubusercontent.com/1050029/140507377-2c98853b-2fcb-44d9-8640-d82381a30d76.png" alt="image" /></p>
<p>There are a few problems that I would love to solve right now, and the most favorite is Professional Networking. LinkedIn is good but is no more relevant anymore.</p>
<p>We first need a professional network that caters to each individual differently, be it a programmer, designer, or salesperson.</p>
<p>The second is the Social aspect of the Professional Network. Is it necessary? And how useful is it?
Whatever may be the answers to these questions, what LinkedIn has is not the solution. Instead, it has become another Facebook, with less and less relevance to Professional Networking.</p>
<p>The third and very critical aspect of professional networks is meaningful connections. Connecting to mutually beneficial people shouldn’t be as hard as it is right now. One can reimagine the whole ecosystem for better, faster, and efficient ways of professionally connecting.</p>
<p>When I was searching for the companies, I came across Peerlist via a tweet from a user. We (Vishwesh and I) quickly got in touch with <a href="https://peerlist.io/designerdada">@Akash</a> and <a href="https://peerlist.io/yogini">@Yogini</a>. When we spoke to them, we understood the zeal they were building this product. After looking at the roadmap, @Vishwesh and I realized that we significantly align on the problems of the current solutions and probable solutions. We believe that Peerlist could be one of the ways of imagining how future Professional Networks should be. And <a href="https://peerlist.io/designerdada">@Akash</a> and <a href="https://peerlist.io/yogini">@Yogini</a>, the company’s co-founders, have the right skills and approach to solving this problem.</p>
<p><a href="https://peerlist.io?ref=amd">Peerlist</a> is launching the platform soon, but till then, please register for the private beta now 🙂.</p>
<blockquote>
<p><strong>If you are starting up and the domain is a matrimonial/dating network, you think we can help. Feel free to reach out to us at <a href="akshay@sifar.vc">akshay@sifar.vc</a> or <a href="vishwesh@sifar.vc">vishwesh@sifar.vc</a></strong></p>
</blockquote>Akshay DeoWhy Angel Investing?💬 Managing Your Slack Instance2020-12-25T13:00:46+00:002020-12-25T13:00:46+00:00https://www.akshaydeo.com/blog/2020/12/25/slack<p>Searchable Log of All Communication and Knowledge (aka your Slack instance) could get chaotic if not managed well. But if managed well, it saves a lot of meetings in your company. And why should you trust me? I was part of Slack for 15 months, and I have used Slack the way Slack uses it.</p>
<h3 id="and-why-is-it-important-to-you">And why is it important to you?</h3>
<p>Managing Companies Slack instances effectively will change many things for you; Less confusion, less chaos, less disturbance, fewer meetings, and more productive time.</p>
<p>Managing your Slack instance has a simple formula, But requires a lot of unlearning. Before going into managing Slack instance, we will see the pros and cons of using Slack in your company.</p>
<h3 id="pros">Pros</h3>
<ul>
<li>Get structured internal communications among channels.</li>
<li>You can use it with other companies (your clients, vendors) using Shared channels.</li>
<li>Integrate all your tools in one place.</li>
</ul>
<h3 id="cons">Cons</h3>
<ul>
<li>If not maintained well, it could create a lot of noise.</li>
<li>Essential messages could get lost in all the chaos.</li>
<li>Everything seems ASAP.</li>
<li>Everyone is working all the time.</li>
</ul>
<h1 id="the-structure-that-works-for-considerably-bigger-companiesteams">The structure that works for considerably bigger companies/teams</h1>
<p>Divide your entire company into multiple teams. So here we are building channels for team “search.”</p>
<p>There will be one channel called <code class="language-plaintext highlighter-rouge">team-search</code>. This will be the place for everyone inside the company interested in knowing what’s happening with Search.
Then there will be team specific channels:</p>
<p><img src="/public/images/slack-management.png" /></p>
<table>
<thead>
<tr>
<th>Channel</th>
<th>Purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">devel-search</code></td>
<td>Channel specific to developers of the team. This involves discussions specific to development flows etc.</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">escal-search</code></td>
<td>Channel specific to escalations. Here you will integrate all your tools like NewRelic, Datadog, Pagerduty, CloudWatch alerts to escalate your service anomalies.</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">prod-search</code></td>
<td>Channel specific to product discussions, requests, etc.</td>
</tr>
</tbody>
</table>
<p>You can extend this naming convention for any other cases like</p>
<ol>
<li>For interview channels: <code class="language-plaintext highlighter-rouge">intw-akshay-deo</code>.</li>
<li>Announcement channels: <code class="language-plaintext highlighter-rouge">announce-search</code>.</li>
</ol>
<p>The second minor rule is: <b>using threads heavily</b>. Unfortunately, Slack’s structure does not force everyone to use threads. But with enough practice, separate conversations can be kept/forced to carry in a separate thread inside a channel. This practice does wonder for managing information inside Slack.</p>
<p>I have successfully replicated this structure across different teams, and it does work in most cases. Initially, it takes time to build the team’s muscle memory, but this is achievable with time and enforcement.</p>
<h1 id="clarifying-the-intent-of-urgency">Clarifying the intent of urgency</h1>
<p>While using Slack, follow the etiquette of explaining the urgency of the answer/task. Something like,</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(Not Urgent) Could someone help me with this?
(ASAP) Server 123 is responding with 500; need immediate attention!!
</code></pre></div></div>
<p>This etiquette gives clarity about how and when to react to everyone inside the channel.</p>
<h1 id="use-user-groups-on-slack">Use User Groups on Slack</h1>
<p>Slack has user-groups like email lists. Use them heavily to convey your messages to a group of people.
Mainly create user groups for the teams. Like @team-search and add everyone who is part of the team. This way, whenever you’re communicating something at the team level, you can use @team-search instead of adding everyone manually (and forgetting someone :D)</p>
<h1 id="use-emojis-for-conveying-the-actions-on-a-message">Use Emojis for conveying the actions on a message.</h1>
<p>Slack (for all the right reasons) does not have read receipts, but they have a fantastic feature of reacting to messages. So “👀” for telling that you have seen the message or “✅” to show you completed the request goes a long way!</p>
<h1 id="workflows-are-gems">Workflows are gems</h1>
<p>Workflows are like Zapier inside your Slack. I usually use these workflows to push the JIRA daily update reminders, Individual OKR update reminders, etc. You can build complex flows with Workflow if it’s required. See here for the details.</p>Akshay DeoSearchable Log of All Communication and Knowledge (aka your Slack instance) could get chaotic if not managed well. But if managed well, it saves a lot of meetings in your company. And why should you trust me? I was part of Slack for 15 months, and I have used Slack the way Slack uses it.How to get better at Engineering Management!2020-09-16T19:00:46+00:002020-09-16T19:00:46+00:00https://www.akshaydeo.com/blog/2020/09/16/principles<p>These are essential principles for being a better engineering manager and setting up better engineering management processes. Most of these principles are from the books that I have mentioned in the introductory post. I deserve very little credit for this.</p>
<h2 id="consistency-is-the-key">Consistency is the key</h2>
<ul>
<li>Everything that we, as managers, do has to be consistent.</li>
<li>Any strategy/policy becomes effective when applied for sufficiently long periods.</li>
<li>So pick the tools you are comfortable with, and implement if you pledge to use them consistently.</li>
</ul>
<h2 id="be-iterative-when-it-comes-to-processes">Be iterative when it comes to processes</h2>
<ul>
<li>Change is the only constant 😌. And every process needs changes and iterations to make it perfect.</li>
<li>Keep these processes open for iterations. Do not lock on something unless you can prove it works. Also, do not engage the entire team from the inception of the processes.</li>
<li>Have <a href="https://medium.com/@ameet/strong-opinions-weakly-held-a-framework-for-thinking-6530d417e364">strong opinions loosely held!</a> You will find many counter-arguments for this philosophy, but this is highly important to keep your approach iterative.</li>
<li>Don’t add exceptions, count the exceptions, and update the process if an exception repeatedly occurs over a considerable amount of time.</li>
</ul>
<h2 id="actionable-feedbacks">Actionable feedbacks</h2>
<ul>
<li>The outcome of each engineering management process should be actionable with little or no manual efforts.
<ul>
<li>For the standup flow mentioned above, each standup message processing should give information about
<ul>
<li>Progress of the project</li>
<li>Blockers that an EM needs to look into on high priority.</li>
</ul>
</li>
</ul>
</li>
<li>Each actionable output should be converted into a metric and tracked throughout the lifetime of the action.
<ul>
<li>Based on the overall progress and blockers list, the system should give an overall health score based on some pre-defined weighting formula.</li>
</ul>
</li>
<li>If it’s too low, the project needs some special attention. 🥋</li>
</ul>
<h2 id="always-complete-the-loop">Always complete the loop</h2>
<ul>
<li>Most of the processes adopted by teams do not complete the loop. So no one can learn from these processes and keep on repeating almost the same mistakes in different forms.</li>
<li>Completing the loop is the job of an engineering manager. And one of the trickiest and tedious one.</li>
<li>For standups, making sure all the blockers are solved, digging deeper if the health score is low are the examples of completing the loop.</li>
</ul>
<h2 id="precisely-over-communicate">Precisely over-communicate</h2>
<ul>
<li>Firmly establish the directions of the culture code/expectations from the team members. And keep on repeating it, even if you think you have done it well in the first go.</li>
<li>Sometimes these conversations become unpleasant, repetitive, and impertinent. But its all worth it.</li>
<li>Aligning everyone from the team towards the common goal with consistent practices and culture code is the recipe for successful execution.</li>
</ul>
<h2 id="open-up-communication-channels">Open up communication channels</h2>
<ul>
<li>Some members, specifically junior members of the team, may not feel comfortable voicing their opinions. Make sure senior members of your team are standing up for junior members.</li>
<li>Force team leads/project leads to create private channels, which are not accessible to the managers/leadership where every person in the group can voice all of their problems.</li>
<li>Never attribute issues to specific people. If there is a problem, there is a problem 😁. And its EMs job to look into it constructively.</li>
</ul>
<h2 id="trust-in-your-teams-capabilities">Trust in your teams capabilities</h2>
<div class="container">
<div class="row">
<div class="col-lg-5 col-12">
<img class="img-thumbnail rounded d-block mx-auto" src="/public/images/productivity.jpeg" style="max-height:200px" />
</div>
<div class="col-lg-7 col-12">
<ul>
<li> More you spend time with the team, you start understanding the strengths and weaknesses your team posses. But take a cons iderable amount of time to build your first capability matrix, ideally 2+ months.</li>
<li>Once you know these capabilities, start doubling down on strengths. Start removing yourself from the decisions which involve these strengths.</li>
<li>And then start focussing on weaknesses. I will speak about it in detail in another post, but the most effective way of fixing this is, bringing the complementing talent in the team. Either from another team or by hiring from outside. </li>
</ul>
</div>
</div>
</div>
<div class="container">
<div class="row">
<div class="col-12 text-center">
<figure class="figure">
<img src="/public/images/capability_matrix.png" class="figure-img img-fluid rounded" alt="..." style="max-height:200px" />
<figcaption class="figure-caption">A traditional Capability matrix for your team.</figcaption>
</figure>
</div>
</div>
</div>
<h2 id="automated-flows">Automated flows</h2>
<ul>
<li>Engineers hate redundancy. And they should!</li>
<li>Ensure that the processes will leverage advanced APIs, ML, and NLP systems to remove the friction.</li>
<li>Long threads of daily standups notes. Everyone from a team pushes what they have worked on, what they will do today and blockers.</li>
<li>These are plain text, without forced formatting on the messages. So it depends on individuals to make it concisely insightful</li>
<li>Team members (not all team members but most of them) have very little insightful information while pushing these updates</li>
</ul>
<div class="info-box" style="margin-left:20px;margin-top:0px;" role="alert">
<h3 class="title">🤔 How can we improve this?</h3>
<hr style="margin-top:10px;margin-bottom:10px" />
<div class="content">
<span>✅ They should be able to see what they had written yesterday in a checklist format to verify what they accomplished yesterday.</span><br />
<span>✅ They should be able to see all the meetings/interviews they have to participate in and different PR reviews pending.</span><br />
<span>🏆 This information will help team members to push data, which is realistic.</span>
</div>
</div>
<ul>
<li>Engineering managers keep on losing track as this information is not easily queryable.</li>
</ul>
<div class="info-box" style="margin-left:20px;margin-top:0px;" role="alert">
<h3 class="title">🤔 How can we improve this?</h3>
<hr style="margin-top:10px;margin-bottom:10px" />
<div class="content">
<span>✅ Parse these standup messages and automatically updated on a dashboard complemented with advanced filtering would be a big help..</span>
</div>
</div>
<ul>
<li>Hence results in repeated information pulling, more status sync-up meetings, which wastes everyone’s time.</li>
<li>Robust and habit forming management flows are built with <strong>Trigger ➡️ Action ➡️ Variable Rewards ➡️ Investment</strong>.
<ul>
<li>Timely triggers are crucial for an automated system.
<ul>
<li>Example Sending a Slack message with an action button to push the Standup update every morning at 10:30 am (local time of the team member).</li>
<li>Worst case, send an email 😃(I love Slack)</li>
</ul>
</li>
<li>Action happens out of Intention. But this intention comes with timely triggers.</li>
<li>Rewards are in terms insightful information, higher visibility, automated sync-ups, less distraction for team members.</li>
<li>Investment is pushing information into these systems to keep on getting rewards.</li>
</ul>
</li>
</ul>
<div class="info-box" style="margin-top:0px;" role="alert">
<h3 class="title">🤔 Things to consider when selecting engineering management tools </h3>
<hr style="margin-top:10px;margin-bottom:10px" />
<div class="content">
<span>✅ Pick one central tool like Slack or Teams, where most of the team is already present.</span><br />
<span>✅ Bring tools that connect with these central tools, so that teams don’t have to switch the context.</span><br />
<span>✅ Tools should give insights to speed up the decision making process for the team.</span><br />
<span>✅ And one important thing: These tools are vital to your job, not to the team. So make sure you don’t expect teams to make massive time and effort investments just to make your job easier.</span>
</div>
</div>Akshay DeoThese are essential principles for being a better engineering manager and setting up better engineering management processes. Most of these principles are from the books that I have mentioned in the introductory post. I deserve very little credit for this.Some Personal News : I am joining Postman 🚀2020-09-11T13:00:46+00:002020-09-11T13:00:46+00:00https://www.akshaydeo.com/blog/2020/09/11/postman<p>APIs are the foundation of modern applications. From complex financial flows to simplest photo-sharing apps breath through APIs. Over the last decade, I’ve seen APIs becoming central to the software development lifecycle.</p>
<p>During this shift, Postman was one of those few companies that understood the importance of building sophisticated workflows around API development. Fast forward eight years, Postman built the entire ecosystem of collaborative API design, development, mocking, monitoring, documentation.</p>
<p>I’ve been using Postman for almost seven years now. And I am still being fascinated by the immense grasp over predicting the future of API development tools. On another side, I’ve always loved the natural progression from a side-project to an actual company.</p>
<p><img class="rounded mx-auto d-block" src="/public/images/pm-orange-logo-horiz.svg" width="60%" style="min-width:150px;" /></p>
<p>When I spoke with Ankit, Abhinav, and the rest of the leadership, I learned the ambitious mission and the systematic approach for achieving it. There are many synergies between the way I think of work and the way Postman works as a company. Watch this presentation by Ankit (CTO) in SAAS conference 2020.</p>
<div class="row" style="margin-top: 20px; margin-bottom: 20px">
<div class="col-sm-12 col-md-6 offset-md-3">
<div style="margin-bottom:30px">
<iframe style="display: block; margin: auto; min-height: 300px; padding: 5px" width="100%" src="https://www.youtube.com/embed/dKg0nb9DOOY" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<i style="font-size:13px">Watch this presentation by Ankit (CTO) in SaaS BOOMi conference 2020.</i>
</div>
</div>
</div>
<p><strong>I am super excited to share that I am going to be the part of the Postman’s engineering team 🎉</strong>. I’ll be starting from 21st September 2020.</p>
<p>I will be focussing on our 3 products</p>
<ol>
<li><a href="https://www.postman.com/features/mock-api/">API Mock and Design</a></li>
<li><a href="https://www.postman.com/api-documentation-tool/">API Documentation</a></li>
<li><a href="https://www.postman.com/api-platform/">API Platform</a></li>
</ol>
<p><strong>Another exciting news is, Postman is going to have a presence in Pune, 🇮🇳</strong>. Postman is principally a remote-first company spread across 5 continents and 7 countries, but we are also open to working from an office. Get in touch with me if you are interested in becoming part of this mission from Pune, Bangalore or Remote 🙇♂️. Here is the <a href="https://www.postman.com/careers/open-positions/">careers</a> page for reference.</p>Akshay DeoAPIs are the foundation of modern applications. From complex financial flows to simplest photo-sharing apps breath through APIs. Over the last decade, I’ve seen APIs becoming central to the software development lifecycle.Securing Electron app source code2019-10-28T13:00:46+00:002019-10-28T13:00:46+00:00https://www.akshaydeo.com/blog/2019/10/28/securing-electron-app-source-code<p><a href="https://electronjs.org">Electron</a> is one of the best ways for building cross-platform desktop apps. When I was evaluating a toolkit for building <a href="https://viwr.app">Viwr</a>, Electron was a clear winner over nw.js and Qt-based on the community, resources and available libraries.</p>
<h2 id="-my-electron-stack">📚 My Electron stack</h2>
<ul>
<li>Electron : 5.0.4</li>
<li>ReactJS : 16.4.1</li>
</ul>
<p>In this codebase, obfuscation and mangling for ReactJS are done by the react-scripts. This blog mainly focuses on obfuscating electron related codebase.</p>
<h3 id="-code-structure">🏗 Code structure</h3>
<p>I am using ReactJS for writing the front-end part of the Electron app. I have not used any scaffolding scripts for this.</p>
<div class="container">
<div class="row">
<div class="col-md-4 col-sm-12">
<img class="img-responsive" src="/public/images/viwr-code-structure.png" width="100%" />
</div>
<div class="col-md-8 col-sm-12">
<h4>src</h4>
<code>src</code> is the entry point for all the source code. This folder contains <code>index.js</code> and <code>index.css</code> which are ReactJS specific.
<h4>electrons</h4>
This folder contains all the BrowserWindow`s and corresponding main thread related code. We will be obfuscating files in this folder.
<h4>components</h4>
This folder contains all the <code>ReactJS</code> components.
</div>
</div>
</div>
<h2 id="-securing-the-source-code">🛡 Securing the source code</h2>
<p>Securing the source code is one of the most important steps before releasing any app (unless it’s an open-source project). There is very little help available on the web regarding obfuscating and mangling the electron source code.</p>
<div class="container">
<div class="row">
<div class="col-12">
<img class="img-responsive" src="/public/images/config.png" width="100%" />
</div>
<div class="col-12 text-center" style="font-size:11px;">
<p>Icons by <a href="https://www.glazestock.com/artist/Nermin">Nermin</a> from <a href="https://glazestock.com">glazestock.com</a></p>
</div>
</div>
</div>
<h3 id="-tools">📦 Tools</h3>
<h4 id="1-craco">1. CRACO</h4>
<blockquote>
<p>This is required only if you are using ReactJS for writing your Electron app.</p>
</blockquote>
<p>Create React App Configuration Override (CRACO) is an easy and comprehensible configuration layer for create-react-app.</p>
<p>Get all the benefits of create-react-app and customization without using ‘eject’ by adding a single craco.config.js file at the root of your application and customize your eslint, babel, postcss configurations and many more.</p>
<div class="container text-center" style="margin-bottom:-5px;"><code>craco.config.js</code></div>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">webpack</span><span class="p">:</span> <span class="p">{</span>
<span class="na">configure</span><span class="p">:</span> <span class="p">(</span><span class="nx">webpackConfig</span><span class="p">,</span> <span class="p">{</span> <span class="nx">env</span><span class="p">,</span> <span class="nx">paths</span> <span class="p">})</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1">// Adding rule for web workers</span>
<span class="nx">webpackConfig</span><span class="p">.</span><span class="nx">module</span><span class="p">.</span><span class="nx">rules</span><span class="p">.</span><span class="nx">push</span><span class="p">({</span>
<span class="na">test</span><span class="p">:</span> <span class="sr">/</span><span class="se">\.</span><span class="sr">worker</span><span class="se">\.(</span><span class="sr">js|jsx|mjs</span><span class="se">)</span><span class="sr">$/</span><span class="p">,</span>
<span class="na">use</span><span class="p">:</span> <span class="p">{</span>
<span class="na">loader</span><span class="p">:</span> <span class="dl">"</span><span class="s2">worker-loader</span><span class="dl">"</span><span class="p">,</span>
<span class="na">options</span><span class="p">:</span> <span class="p">{</span>
<span class="na">fallback</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="na">inline</span><span class="p">:</span> <span class="kc">true</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="nx">webpackConfig</span><span class="p">.</span><span class="nx">target</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">electron-renderer</span><span class="dl">"</span><span class="p">;</span>
<span class="nx">webpackConfig</span><span class="p">[</span><span class="dl">"</span><span class="s2">output</span><span class="dl">"</span><span class="p">][</span><span class="dl">"</span><span class="s2">globalObject</span><span class="dl">"</span><span class="p">]</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">this</span><span class="dl">"</span><span class="p">;</span>
<span class="nx">webpackConfig</span><span class="p">[</span><span class="dl">"</span><span class="s2">optimization</span><span class="dl">"</span><span class="p">][</span><span class="dl">"</span><span class="s2">noEmitOnErrors</span><span class="dl">"</span><span class="p">]</span> <span class="o">=</span> <span class="kc">false</span><span class="p">;</span>
<span class="nx">webpackConfig</span><span class="p">[</span><span class="dl">"</span><span class="s2">node</span><span class="dl">"</span><span class="p">][</span><span class="dl">"</span><span class="s2">fs</span><span class="dl">"</span><span class="p">]</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">empty</span><span class="dl">"</span><span class="p">;</span>
<span class="k">return</span> <span class="nx">webpackConfig</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">};</span>
</code></pre></div></div>
<p>This script</p>
<ul>
<li>Adds a worker-loader for the renderer thread flow.</li>
<li>Sets up the target as electron-renderer.</li>
<li>Enables <code>fs</code> in renderer-thread land.</li>
</ul>
<h4 id="2-webpack">2. Webpack</h4>
<p>An all in one tool for bundling any javascript project. I used a webpack script for merging all my electron main thread codebase into one single object.</p>
<div class="container text-center" style="margin-bottom:-5px;"><code>webpack.config.js</code></div>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">path</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">"</span><span class="s2">path</span><span class="dl">"</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">fs</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">"</span><span class="s2">fs</span><span class="dl">"</span><span class="p">);</span>
<span class="kd">let</span> <span class="nx">node_modules</span> <span class="o">=</span> <span class="p">{};</span>
<span class="nx">fs</span><span class="p">.</span><span class="nx">readdirSync</span><span class="p">(</span><span class="dl">"</span><span class="s2">node_modules</span><span class="dl">"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">filter</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">x</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">[</span><span class="dl">"</span><span class="s2">.bin</span><span class="dl">"</span><span class="p">].</span><span class="nx">indexOf</span><span class="p">(</span><span class="nx">x</span><span class="p">)</span> <span class="o">===</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
<span class="p">})</span>
<span class="p">.</span><span class="nx">forEach</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">mod</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">node_modules</span><span class="p">[</span><span class="nx">mod</span><span class="p">]</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">commonjs </span><span class="dl">"</span> <span class="o">+</span> <span class="nx">mod</span><span class="p">;</span>
<span class="p">});</span>
<span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">mode</span><span class="p">:</span> <span class="dl">"</span><span class="s2">production</span><span class="dl">"</span><span class="p">,</span>
<span class="na">entry</span><span class="p">:</span> <span class="dl">"</span><span class="s2">./src/electrons/main.js</span><span class="dl">"</span><span class="p">,</span>
<span class="na">output</span><span class="p">:</span> <span class="p">{</span>
<span class="na">pathinfo</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">path</span><span class="p">:</span> <span class="nx">path</span><span class="p">.</span><span class="nx">resolve</span><span class="p">(</span><span class="o">**</span><span class="nx">dirname</span><span class="p">,</span> <span class="dl">"</span><span class="s2">build</span><span class="dl">"</span><span class="p">),</span>
<span class="na">filename</span><span class="p">:</span> <span class="dl">"</span><span class="s2">electrons.raw.js</span><span class="dl">"</span><span class="p">,</span>
<span class="na">globalObject</span><span class="p">:</span> <span class="dl">"</span><span class="s2">this</span><span class="dl">"</span>
<span class="p">},</span>
<span class="na">externals</span><span class="p">:</span> <span class="nx">node_modules</span><span class="p">,</span>
<span class="na">node</span><span class="p">:</span> <span class="p">{</span>
<span class="o">**</span><span class="na">filename</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="err">\</span><span class="nx">_</span><span class="err">\</span><span class="na">_dirname</span><span class="p">:</span> <span class="kc">false</span>
<span class="p">},</span>
<span class="na">stats</span><span class="p">:</span> <span class="dl">"</span><span class="s2">errors-only</span><span class="dl">"</span><span class="p">,</span>
<span class="na">target</span><span class="p">:</span> <span class="dl">"</span><span class="s2">node</span><span class="dl">"</span>
<span class="p">};</span>
</code></pre></div></div>
<p>This script</p>
<ul>
<li>Combines all the electron main thread code (all the BrowserWindows), and merge them into one single file called <code>electrons.raw.js</code>.</li>
<li>The obfuscation part for renderer thread is handled by <code>react-scripts</code>.</li>
</ul>
<h4 id="3-terser">3. Terser</h4>
<p>A JavaScript parser and mangler/compressor toolkit for ES6+.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">terser --compress --drop_console=true --mangle -- build/electrons.raw.js ></span><span class="w"> </span>build/electrons.js
</code></pre></div></div>
<p>This command</p>
<ul>
<li>Obfuscated and mangles the single merged file we have created using webpack. The final output looks something like</li>
</ul>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">!</span><span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">){</span><span class="kd">var</span> <span class="nx">t</span><span class="o">=</span><span class="p">{};</span><span class="kd">function</span> <span class="nx">n</span><span class="p">(</span><span class="nx">r</span><span class="p">){</span><span class="k">if</span><span class="p">(</span><span class="nx">t</span><span class="p">[</span><span class="nx">r</span><span class="p">])</span><span class="k">return</span> <span class="nx">t</span><span class="p">[</span><span class="nx">r</span><span class="p">].</span><span class="nx">exports</span><span class="p">;</span><span class="kd">var</span> <span class="nx">i</span><span class="o">=</span><span class="nx">t</span><span class="p">[</span><span class="nx">r</span><span class="p">]</span><span class="o">=</span><span class="p">{</span><span class="na">i</span><span class="p">:</span><span class="nx">r</span><span class="p">,</span><span class="na">l</span><span class="p">:</span><span class="o">!</span><span class="mi">1</span><span class="p">,</span><span class="na">exports</span><span class="p">:{}};</span><span class="k">return</span> <span class="nx">e</span><span class="p">[</span><span class="nx">r</span><span class="p">].</span><span class="nx">call</span><span class="p">(</span><span class="nx">i</span><span class="p">.</span><span class="nx">exports</span><span class="p">,</span><span class="nx">i</span><span class="p">,</span><span class="nx">i</span><span class="p">.</span><span class="nx">exports</span><span class="p">,</span><span class="nx">n</span><span class="p">),</span><span class="nx">i</span><span class="p">.</span><span class="nx">l</span><span class="o">=!</span><span class="mi">0</span><span class="p">,</span><span class="nx">i</span><span class="p">.</span><span class="nx">exports</span><span class="p">}</span><span class="nx">n</span><span class="p">.</span><span class="nx">m</span><span class="o">=</span><span class="nx">e</span><span class="p">,</span><span class="nx">n</span><span class="p">.</span><span class="nx">c</span><span class="o">=</span><span class="nx">t</span><span class="p">,</span><span class="nx">n</span><span class="p">.</span><span class="nx">d</span><span class="o">=</span><span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">,</span><span class="nx">t</span><span class="p">,</span><span class="nx">r</span><span class="p">){</span><span class="nx">n</span><span class="p">.</span><span class="nx">o</span><span class="p">(</span><span class="nx">e</span><span class="p">,</span><span class="nx">t</span><span class="p">)</span><span class="o">||</span><span class="nb">Object</span><span class="p">.</span><span class="nx">defineProperty</span><span class="p">(</span><span class="nx">e</span><span class="p">,</span><span class="nx">t</span><span class="p">,{</span><span class="na">enumerable</span><span class="p">:</span><span class="o">!</span><span class="mi">0</span><span class="p">,</span><span class="na">get</span><span class="p">:</span><span class="nx">r</span><span class="p">})},</span><span class="nx">n</span><span class="p">.</span><span class="nx">r</span><span class="o">=</span><span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">){</span><span class="dl">"</span><span class="s2">undefined</span><span class="dl">"</span><span class="o">!=</span><span class="k">typeof</span> <span class="nb">Symbol</span><span class="o">&&</span><span class="nb">Symbol</span><span class="p">.</span><span class="nx">toStringTag</span><span class="o">&&</span><span class="nb">Object</span><span class="p">.</span><span class="nx">defineProperty</span><span class="p">(</span><span class="nx">e</span><span class="p">,</span><span class="nb">Symbol</span><span class="p">.</span><span class="nx">toStringTag</span><span class="p">,{</span><span class="na">value</span><span class="p">:</span><span class="dl">"</span><span class="s2">Module</span><span class="dl">"</span><span class="p">}),</span><span class="nb">Object</span><span class="p">.</span><span class="nx">defineProperty</span><span class="p">(</span><span class="nx">e</span><span class="p">,</span><span class="dl">"</span><span class="s2">__esModule</span><span class="dl">"</span><span class="p">,{</span><span class="na">value</span><span class="p">:</span><span class="o">!</span><span class="mi">0</span><span class="p">})},</span><span class="nx">n</span><span class="p">.</span><span class="nx">t</span><span class="o">=</span><span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">,</span><span class="nx">t</span><span class="p">){</span><span class="k">if</span><span class="p">(</span><span class="mi">1</span><span class="o">&</span><span class="nx">t</span><span class="o">&&</span><span class="p">(</span><span class="nx">e</span><span class="o">=</span><span class="nx">n</span><span class="p">(</span><span class="nx">e</span><span class="p">)),</span><span class="mi">8</span><span class="o">&</span><span class="nx">t</span><span class="p">)</span><span class="k">return</span> <span class="nx">e</span><span class="p">;</span><span class="k">if</span><span class="p">(</span><span class="mi">4</span><span class="o">&</span><span class="nx">t</span><span class="o">&&</span><span class="dl">"</span><span class="s2">object</span><span class="dl">"</span><span class="o">==</span><span class="k">typeof</span> <span class="nx">e</span><span class="o">&&</span><span class="nx">e</span><span class="o">&&</span><span class="nx">e</span><span class="p">.</span><span class="nx">__esModule</span><span class="p">)</span><span class="k">return</span> <span class="nx">e</span><span class="p">;</span><span class="kd">var</span> <span class="nx">r</span><span class="o">=</span><span class="nb">Object</span><span class="p">.</span><span class="nx">create</span><span class="p">(</span><span class="kc">null</span><span class="p">);</span><span class="k">if</span><span class="p">(</span><span class="nx">n</span><span class="p">.</span><span class="nx">r</span><span class="p">(</span><span class="nx">r</span><span class="p">),</span><span class="nb">Object</span><span class="p">.</span><span class="nx">defineProperty</span><span class="p">(</span><span class="nx">r</span><span class="p">,</span><span class="dl">"</span><span class="s2">default</span><span class="dl">"</span><span class="p">,{</span><span class="na">enumerable</span><span class="p">:</span><span class="o">!</span><span class="mi">0</span><span class="p">,</span><span class="na">value</span><span class="p">:</span><span class="nx">e</span><span class="p">}),</span><span class="mi">2</span><span class="o">&</span><span class="nx">t</span><span class="o">&&</span><span class="dl">"</span><span class="s2">string</span><span class="dl">"</span><span class="o">!=</span><span class="k">typeof</span> <span class="nx">e</span><span class="p">)</span><span class="k">for</span><span class="p">(</span><span class="kd">var</span> <span class="nx">i</span> <span class="k">in</span> <span class="nx">e</span><span class="p">)</span><span class="nx">n</span><span class="p">.</span><span class="nx">d</span><span class="p">(</span><span class="nx">r</span><span class="p">,</span><span class="nx">i</span><span class="p">,</span><span class="kd">function</span><span class="p">(</span><span class="nx">t</span><span class="p">){</span><span class="k">return</span> <span class="nx">e</span><span class="p">[</span><span class="nx">t</span><span class="p">]}.</span><span class="nx">bind</span><span class="p">(</span><span class="kc">null</span><span class="p">,</span><span class="nx">i</span><span class="p">));</span><span class="k">return</span> <span class="nx">r</span><span class="p">},</span><span class="nx">n</span><span class="p">.</span><span class="nx">n</span><span class="o">=</span><span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">){</span><span class="kd">var</span> <span class="nx">t</span><span class="o">=</span><span class="nx">e</span><span class="o">&&</span><span class="nx">e</span><span class="p">.</span><span class="nx">__esModule</span><span class="p">?</span><span class="kd">function</span><span class="p">(){</span><span class="k">return</span> <span class="nx">e</span><span class="p">.</span><span class="k">default</span><span class="p">}:</span><span class="kd">function</span><span class="p">(){</span><span class="k">return</span> <span class="nx">e</span><span class="p">};</span><span class="k">return</span> <span class="nx">n</span><span class="p">.</span><span class="nx">d</span><span class="p">(</span><span class="nx">t</span><span class="p">,</span><span class="dl">"</span><span class="s2">a</span><span class="dl">"</span><span class="p">,</span><span class="nx">t</span><span class="p">),</span><span class="nx">t</span><span class="p">},</span><span class="nx">n</span><span class="p">.</span><span class="nx">o</span><span class="o">=</span><span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">,</span><span class="nx">t</span><span class="p">){</span><span class="k">return</span> <span class="nb">Object</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">hasOwnProperty</span><span class="p">.</span><span class="nx">call</span><span class="p">(</span><span class="nx">e</span><span class="p">,</span><span class="nx">t</span><span class="p">)},</span><span class="nx">n</span><span class="p">.</span><span class="nx">p</span><span class="o">=</span><span class="dl">""</span><span class="p">,</span><span class="nx">n</span><span class="p">(</span><span class="nx">n</span><span class="p">.</span><span class="nx">s</span><span class="o">=</span><span class="mi">63</span><span class="p">)}([</span><span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">,</span><span class="nx">t</span><span class="p">){</span><span class="nx">e</span><span class="p">.</span><span class="nx">exports</span><span class="o">=</span><span class="nx">require</span><span class="p">(</span><span class="dl">"</span><span class="s2">tslib</span><span class="dl">"</span><span class="p">)},</span><span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">,</span><span class="nx">t</span><span class="p">){</span><span class="nx">e</span><span class="p">.</span><span class="nx">exports</span><span class="o">=</span><span class="nx">require</span><span class="p">(</span><span class="dl">"</span><span class="s2">electron</span><span class="dl">"</span><span class="p">)},</span><span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">,</span><span class="nx">t</span><span class="p">,</span><span class="nx">n</span><span class="p">){</span><span class="dl">"</span><span class="s2">use strict</span><span class="dl">"</span><span class="p">;</span><span class="nb">Object</span><span class="p">.</span><span class="nx">defineProperty</span><span class="p">(</span><span class="nx">t</span><span class="p">,</span><span class="dl">"</span><span class="s2">__esModule</span><span class="dl">"</span><span class="p">,{</span><span class="na">value</span><span class="p">:</span><span class="o">!</span><span class="mi">0</span><span class="p">});</span><span class="kd">var</span> <span class="nx">r</span><span class="o">=</span><span class="nx">n</span><span class="p">(</span><span class="mi">34</span><span class="p">);</span><span class="nx">t</span><span class="p">.</span><span class="nx">addBreadcrumb</span><span class="o">=</span><span class="nx">r</span><span class="p">.</span><span class="nx">addBreadcrumb</span><span class="p">,</span><span class="nx">t</span><span class="p">.</span><span class="nx">captureException</span><span class="o">=</span><span class="nx">r</span><span class="p">.</span><span class="nx">captureException</span><span class="p">,</span><span class="nx">t</span><span class="p">.</span><span class="nx">captureEvent</span><span class="o">=</span><span class="nx">r</span><span class="p">.</span><span class="nx">captureEvent</span><span class="p">,</span>
</code></pre></div></div>
<h2 id="script-for-building-the-codebase">Script for building the codebase</h2>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">RED</span><span class="o">=</span><span class="s1">'\033[0;31m'</span>
<span class="nv">GREEN</span><span class="o">=</span><span class="s1">'\033[0;32m'</span>
<span class="nv">NC</span><span class="o">=</span><span class="s1">'\033[0m'</span>
<span class="nb">rm</span> <span class="nt">-r</span> dist <span class="o">||</span> <span class="nb">true</span> <span class="o">&&</span> <span class="nb">rm</span> <span class="nt">-r</span> build <span class="o">||</span> <span class="nb">true</span>
<span class="c"># To build reactjs</span>
yarn build
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$?</span> <span class="nt">-ne</span> 0 <span class="o">]]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">RED</span><span class="k">}</span><span class="s2">BUILD FAILED</span><span class="k">${</span><span class="nv">NC</span><span class="k">}</span><span class="s2"> => </span><span class="nv">$?</span><span class="s2">"</span>
<span class="nb">exit</span> <span class="nt">-1</span>
<span class="k">fi</span>
<span class="c"># To bundle the code</span>
yarn bundle
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$?</span> <span class="nt">-ne</span> 0 <span class="o">]]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">RED</span><span class="k">}</span><span class="s2">BUNDLING FAILED</span><span class="k">${</span><span class="nv">NC</span><span class="k">}</span><span class="s2"> => </span><span class="nv">$?</span><span class="s2">"</span>
<span class="nb">exit</span> <span class="nt">-1</span>
<span class="k">fi</span>
<span class="c"># Obfuscating the code</span>
terser <span class="nt">--compress</span> <span class="nt">--drop_console</span><span class="o">=</span><span class="nb">true</span> <span class="nt">--mangle</span> <span class="nt">--</span> build/electrons.raw.js <span class="o">></span> build/electrons.js
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$?</span> <span class="nt">-ne</span> 0 <span class="o">]]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">RED</span><span class="k">}</span><span class="s2">OBFUSCATION FAILED</span><span class="k">${</span><span class="nv">NC</span><span class="k">}</span><span class="s2"> => </span><span class="nv">$?</span><span class="s2">"</span>
<span class="nb">exit</span> <span class="nt">-1</span>
<span class="k">fi</span>
<span class="c"># Final packaging electron-builder</span>
electron-builder <span class="nt">--config</span>.extraMetadata.main<span class="o">=</span>build/electrons.js
<span class="nv">resp</span><span class="o">=</span><span class="nv">$?</span>
<span class="nb">echo</span> <span class="nv">$resp</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$resp</span> <span class="nt">-ne</span> 0 <span class="o">]]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">RED</span><span class="k">}</span><span class="s2">PACKING FAILED</span><span class="k">${</span><span class="nv">NC</span><span class="k">}</span><span class="s2"> => </span><span class="nv">$resp</span><span class="s2">"</span>
<span class="nb">exit</span> <span class="nt">-1</span>
<span class="k">fi</span>
</code></pre></div></div>
<p>🙇♂️ Thanks</p>Akshay DeoElectron is one of the best ways for building cross-platform desktop apps. When I was evaluating a toolkit for building Viwr, Electron was a clear winner over nw.js and Qt-based on the community, resources and available libraries.How did I improve latency by 700% using sync.Pool2017-12-23T13:25:46+00:002017-12-23T13:25:46+00:00https://www.akshaydeo.com/blog/2017/12/23/How-did-I-improve-latency-by-700-percent-using-syncPool<p>We @media.net write superfast backends with at max 30-40ms turn-around time from web-service. We continuously try to reduce money spent per request. This blog enlists a few of our findings.</p>
<blockquote>
<p>Precautionary warning: It’s a long post with a lot of code. Comment if I am unclear, abstract or vague at any point.</p>
</blockquote>
<h2 id="why-to-use-syncpool">Why to use <code class="language-plaintext highlighter-rouge">sync.Pool</code>?</h2>
<p>sync.Pool [1] comes really handy when one wants to reduce the number of allocations happening during the course of a functionality written in Golang. A Pool is a set of temporary objects that may be individually saved and retrieved. fasthttp [2], zerolog [3] are couple of those most popuplar open source Golang libraries which uses <code class="language-plaintext highlighter-rouge">sync.Pool</code> at the core of their implementation.</p>
<h2 id="how-to-use-syncpool">How to use <code class="language-plaintext highlighter-rouge">sync.Pool</code>?</h2>
<p>Consider the following example:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="p">(</span>
<span class="s">"fmt"</span>
<span class="s">"sync"</span>
<span class="s">"time"</span>
<span class="p">)</span>
<span class="c">// Pool for our struct A</span>
<span class="k">var</span> <span class="n">pool</span> <span class="o">*</span><span class="n">sync</span><span class="o">.</span><span class="n">Pool</span>
<span class="c">// A dummy struct with a member</span>
<span class="k">type</span> <span class="n">A</span> <span class="k">struct</span> <span class="p">{</span>
<span class="n">Name</span> <span class="kt">string</span>
<span class="p">}</span>
<span class="c">// Func to init pool</span>
<span class="k">func</span> <span class="n">initPool</span><span class="p">()</span> <span class="p">{</span>
<span class="n">pool</span> <span class="o">=</span> <span class="o">&</span><span class="n">sync</span><span class="o">.</span><span class="n">Pool</span> <span class="p">{</span>
<span class="n">New</span><span class="o">:</span> <span class="k">func</span><span class="p">()</span><span class="k">interface</span><span class="p">{}</span> <span class="p">{</span>
<span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="s">"Returning new A"</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">new</span><span class="p">(</span><span class="n">A</span><span class="p">)</span>
<span class="p">},</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c">// Main func</span>
<span class="k">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span>
<span class="c">// Initializing pool</span>
<span class="n">initPool</span><span class="p">()</span>
<span class="c">// Get hold of instance one</span>
<span class="n">one</span> <span class="o">:=</span> <span class="n">pool</span><span class="o">.</span><span class="n">Get</span><span class="p">()</span><span class="o">.</span><span class="p">(</span><span class="o">*</span><span class="n">A</span><span class="p">)</span>
<span class="n">one</span><span class="o">.</span><span class="n">Name</span> <span class="o">=</span> <span class="s">"first"</span>
<span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"one.Name = %s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">one</span><span class="o">.</span><span class="n">Name</span><span class="p">)</span>
<span class="c">// Submit back the instance after using</span>
<span class="n">pool</span><span class="o">.</span><span class="n">Put</span><span class="p">(</span><span class="n">one</span><span class="p">)</span>
<span class="c">// Now the same instance becomes usable by another routine without allocating it again</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="benchmarks">Benchmarks</h2>
<p>I picked one of the most commonly used API from our web-service which</p>
<ul>
<li>Performs 4 cache queries</li>
<li>Sends 5 separate events onto Kafka</li>
<li>Calls 2 separate micro-services synchronously</li>
</ul>
<p>I ran a load test of 10-rps on my local machine for 10 seconds first on pooled-flow and then on non-pooled flow.</p>
<blockquote>
<p>These latencies are higher because this service was running in Pune (India) office and communicating with other microservices running in Oregon (United States) :).</p>
</blockquote>
<h3 id="for-pooled-version">For pooled version</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Requests [total, rate] 100, 10.10
Duration [total, attack, wait] 10.189033566s, 9.899999979s, 289.033587ms
Latencies [mean, 50, 95, 99, max] 241.952809ms, 252.716311ms, 476.400585ms, 491.370124ms, 503.938792ms
Bytes In [total, mean] 8400, 84.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:100
Error Set:
</code></pre></div></div>
<h3 id="for-non-pooled-version">For non-pooled version</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Requests [total, rate] 100, 10.10
Duration [total, attack, wait] 13.28824244s, 9.899999931s, 3.388242509s
Latencies [mean, 50, 95, 99, max] 880.57307ms, 325.619739ms, 3.476523432s, 3.685206241s, 3.888228236s
Bytes In [total, mean] 8400, 84.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:100
</code></pre></div></div>
<h3 id="observations">Observations</h3>
<ul>
<li><strong>We get almost 700x performance boost. (95th percentile)</strong></li>
<li>For most of the objects, the ratio of allocation to reuse was almost 10-15 times. Which means the object was allocated probably 5 times and being used almost 50-70 times, during the course of this benchmark.</li>
</ul>
<h3 id="here-are-some-of-the-outcomes-of-one-of-our-services">Here are some of the outcomes of one of our services</h3>
<ul>
<li>Almost 78% of the objects were reused in course of a load test</li>
<li>During the distributed load, this number will improve a lot</li>
</ul>
<p><img src="https://raw.githubusercontent.com/akshaydeo/blog/master/public/images/stats_1.png" alt="img4" />
<img src="https://raw.githubusercontent.com/akshaydeo/blog/master/public/images/stats_2.png" alt="img5" /></p>
<hr />
<h1 id="precautions-to-be-taken-while-using-syncpool">Precautions to be taken while using sync.Pool</h1>
<h3 id="what-if-you-pass-a-pooled-object-to-a-go-routine-and-submit-it-back-to-the-pool-before-go-routine-yet-to-complete-the-execution">What if you pass a pooled object to a go-routine and submit it back to the pool before go-routine yet to complete the execution?</h3>
<p>This is a pickle, once you put back the object, it’s ready for reuse. So before you putting back the object inside the pool, you have to be sure that nothing else is using the same object for any other operation.</p>
<p>Example code - <a href="https://play.golang.org/p/64SoX7W-x1H">play.golang.org</a></p>
<h3 id="lets-consider-this-flow-in-a-more-practical-use-case">Let’s consider this flow in a more practical use case</h3>
<p>Here is the typical flow for a Web-Service API.</p>
<blockquote>
<p>Let’s assume that our controller is using pooled struct to read body of the incoming request.</p>
</blockquote>
<ol>
<li>Go server accepts the request, spawns a go-routine and calls your controller. Typically a func like <code class="language-plaintext highlighter-rouge">func HandleRequest(w http.ResponseWriter, r *http.Request)</code>.</li>
<li>Let’s consider it’s a POST API call, and we read the body of the request, map it to a struct A.</li>
<li>Now through the course of flow, we might do some of the operations on the object in async manner, in a separate go-routine.</li>
<li>At the end of controller functionality, we put back the object into the pool for reuse.</li>
</ol>
<p>Refer the following diagram showing the flow:</p>
<p><img src="https://raw.githubusercontent.com/akshaydeo/blog/master/public/images/go_pooling_image_2.png" alt="img2" /></p>
<p><strong>The issue here is</strong>, what if Async Call 1 takes more time and before it completes it’s functionality, your controller has written the response, and object is put back into the pool!</p>
<h3 id="solution">Solution</h3>
<p>The only solution to it is manual reference counting for the pooled objects. When I started googling about the solution, I came across a nicely written blog here[4].</p>
<p>Basic flow for Reference Counting will be:</p>
<ul>
<li>Increment the counter at the time of Acquiring the object</li>
<li>Decrement the counter once the routine is done using it</li>
<li>Increment the counter before passing the object inside a go-routine or channel</li>
<li>Decrement the counter once the routine is done with processing the object</li>
<li>An object is only put back into the pool when the reference count is zero.</li>
</ul>
<h3 id="implementation">Implementation</h3>
<blockquote>
<p>This implementation is highly inspired from the hydrogen18 blog, but has some modifications which are necessary for a practical web app.</p>
</blockquote>
<p>We will have an interface to have a blueprint of how a Reference Coutable object will be implemented</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// Interface following reference countable interface</span>
<span class="c">// We have provided inbuilt embeddable implementation of the reference countable pool</span>
<span class="c">// This interface just provides the extensibility for the implementation</span>
<span class="k">type</span> <span class="n">ReferenceCountable</span> <span class="k">interface</span> <span class="p">{</span>
<span class="c">// Method to set the current instance</span>
<span class="n">SetInstance</span><span class="p">(</span><span class="n">i</span> <span class="k">interface</span><span class="p">{})</span>
<span class="c">// Method to increment the reference count</span>
<span class="n">IncrementReferenceCount</span><span class="p">()</span>
<span class="c">// Method to decrement reference count</span>
<span class="n">DecrementReferenceCount</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<ul>
<li>IncrementReferenceCount to increase the reference count of the current object</li>
<li>DecrementReferenceCount to decrease the reference count of the current object</li>
</ul>
<hr />
<p>We will write a struct that implements this interface, which can be embedded in our own structs.</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// Struct representing reference</span>
<span class="c">// This struct is supposed to be embedded inside the object to be pooled</span>
<span class="c">// Along with that incrementing and decrementing the references is highly important specifically around routines</span>
<span class="k">type</span> <span class="n">ReferenceCounter</span> <span class="k">struct</span> <span class="p">{</span>
<span class="n">count</span> <span class="o">*</span><span class="kt">uint32</span> <span class="s">`sql:"-" json:"-" yaml:"-"`</span>
<span class="n">destination</span> <span class="o">*</span><span class="n">sync</span><span class="o">.</span><span class="n">Pool</span> <span class="s">`sql:"-" json:"-" yaml:"-"`</span>
<span class="n">released</span> <span class="o">*</span><span class="kt">uint32</span> <span class="s">`sql:"-" json:"-" yaml:"-"`</span>
<span class="n">Instance</span> <span class="k">interface</span><span class="p">{}</span> <span class="s">`sql:"-" json:"-" yaml:"-"`</span>
<span class="n">reset</span> <span class="k">func</span><span class="p">(</span><span class="k">interface</span><span class="p">{})</span> <span class="kt">error</span> <span class="s">`sql:"-" json:"-" yaml:"-"`</span>
<span class="n">id</span> <span class="kt">uint32</span> <span class="s">`sql:"-" json:"-" yaml:"-"`</span>
<span class="p">}</span>
<span class="c">// Method to increment a reference</span>
<span class="k">func</span> <span class="p">(</span><span class="n">r</span> <span class="n">ReferenceCounter</span><span class="p">)</span> <span class="n">IncrementReferenceCount</span><span class="p">()</span> <span class="p">{</span>
<span class="n">atomic</span><span class="o">.</span><span class="n">AddUint32</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">count</span><span class="p">,</span> <span class="m">1</span><span class="p">)</span>
<span class="p">}</span>
<span class="c">// Method to decrement a reference</span>
<span class="c">// If the reference count goes to zero, the object is put back inside the pool</span>
<span class="k">func</span> <span class="p">(</span><span class="n">r</span> <span class="n">ReferenceCounter</span><span class="p">)</span> <span class="n">DecrementReferenceCount</span><span class="p">()</span> <span class="p">{</span>
<span class="k">if</span> <span class="n">atomic</span><span class="o">.</span><span class="n">LoadUint32</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">count</span><span class="p">)</span> <span class="o">==</span> <span class="m">0</span> <span class="p">{</span>
<span class="nb">panic</span><span class="p">(</span><span class="s">"this should not happen =>"</span> <span class="o">+</span> <span class="n">reflect</span><span class="o">.</span><span class="n">TypeOf</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">Instance</span><span class="p">)</span><span class="o">.</span><span class="n">String</span><span class="p">())</span>
<span class="p">}</span>
<span class="k">if</span> <span class="n">atomic</span><span class="o">.</span><span class="n">AddUint32</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">count</span><span class="p">,</span> <span class="o">^</span><span class="kt">uint32</span><span class="p">(</span><span class="m">0</span><span class="p">))</span> <span class="o">==</span> <span class="m">0</span> <span class="p">{</span>
<span class="n">atomic</span><span class="o">.</span><span class="n">AddUint32</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">released</span><span class="p">,</span> <span class="m">1</span><span class="p">)</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">r</span><span class="o">.</span><span class="n">reset</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">Instance</span><span class="p">);</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
<span class="nb">panic</span><span class="p">(</span><span class="s">"error while resetting an instance => "</span> <span class="o">+</span> <span class="n">err</span><span class="o">.</span><span class="n">Error</span><span class="p">())</span>
<span class="p">}</span>
<span class="n">r</span><span class="o">.</span><span class="n">destination</span><span class="o">.</span><span class="n">Put</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">Instance</span><span class="p">)</span>
<span class="n">r</span><span class="o">.</span><span class="n">instance</span> <span class="o">=</span> <span class="no">nil</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c">// Method to set the current instance</span>
<span class="k">func</span> <span class="p">(</span><span class="n">r</span> <span class="o">*</span><span class="n">ReferenceCounter</span><span class="p">)</span> <span class="n">SetInstance</span><span class="p">(</span><span class="n">i</span> <span class="k">interface</span><span class="p">{})</span> <span class="p">{</span>
<span class="n">r</span><span class="o">.</span><span class="n">instance</span> <span class="o">=</span> <span class="n">i</span>
<span class="p">}</span>
</code></pre></div></div>
<ul>
<li>Reference counter specifically does the job of thread safe reference counting</li>
<li>Along with that, it puts back the instance associated with the counter into the pool as soon as the reference count becomes zero</li>
<li>Reset function ideally resets all the members of the instance of an object.</li>
<li>
<p>This is necessary in most of the practical use cases</p>
<ul>
<li>Consider the following struct</li>
</ul>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">type</span> <span class="n">A</span> <span class="k">struct</span><span class="p">{</span>
<span class="n">First</span> <span class="kt">string</span> <span class="s">`json:"f"`</span>
<span class="n">Second</span> <span class="kt">string</span> <span class="s">`json:"s"`</span>
<span class="n">Third</span> <span class="kt">int</span> <span class="s">`json:"t"`</span>
<span class="p">}</span>
</code></pre></div> </div>
<ul>
<li>Suppose that while using for the first time we got a JSON like following</li>
</ul>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"f"</span><span class="p">:</span><span class="w"> </span><span class="s2">"one"</span><span class="p">,</span><span class="w">
</span><span class="nl">"t"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div> </div>
<ul>
<li>We do the processing and put back the object inside the pool.</li>
<li>Now we got the second request with following JSON</li>
</ul>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"s"</span><span class="p">:</span><span class="w"> </span><span class="s2">"second"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div> </div>
<ul>
<li>Technically the resulting struct should have First as “”, Third as 0. But as we are using pooled objects, First will be “one” and Third will be 1.</li>
<li>Hence we use a <code class="language-plaintext highlighter-rouge">Reset</code> function that reset all the members of the object before puting it back in the pool.</li>
</ul>
</li>
</ul>
<hr />
<p>And finally the pool, Reference coutable pool</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// Struct representing the pool</span>
<span class="k">type</span> <span class="n">referenceCountedPool</span> <span class="k">struct</span> <span class="p">{</span>
<span class="n">pool</span> <span class="o">*</span><span class="n">sync</span><span class="o">.</span><span class="n">Pool</span>
<span class="n">factory</span> <span class="k">func</span><span class="p">()</span> <span class="n">ReferenceCountable</span>
<span class="n">returned</span> <span class="kt">uint32</span>
<span class="n">allocated</span> <span class="kt">uint32</span>
<span class="n">referenced</span> <span class="kt">uint32</span>
<span class="p">}</span>
<span class="c">// Method to create a new pool</span>
<span class="k">func</span> <span class="n">NewReferenceCountedPool</span><span class="p">(</span><span class="n">factory</span> <span class="k">func</span><span class="p">(</span><span class="n">referenceCounter</span> <span class="n">ReferenceCounter</span><span class="p">)</span> <span class="n">ReferenceCountable</span><span class="p">,</span> <span class="n">reset</span> <span class="k">func</span><span class="p">(</span><span class="k">interface</span><span class="p">{})</span> <span class="kt">error</span><span class="p">)</span> <span class="o">*</span><span class="n">referenceCountedPool</span> <span class="p">{</span>
<span class="n">p</span> <span class="o">:=</span> <span class="nb">new</span><span class="p">(</span><span class="n">referenceCountedPool</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">pool</span> <span class="o">=</span> <span class="nb">new</span><span class="p">(</span><span class="n">sync</span><span class="o">.</span><span class="n">Pool</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">pool</span><span class="o">.</span><span class="n">New</span> <span class="o">=</span> <span class="k">func</span><span class="p">()</span> <span class="k">interface</span><span class="p">{}</span> <span class="p">{</span>
<span class="c">// Incrementing allocated count</span>
<span class="n">atomic</span><span class="o">.</span><span class="n">AddUint32</span><span class="p">(</span><span class="o">&</span><span class="n">p</span><span class="o">.</span><span class="n">allocated</span><span class="p">,</span> <span class="m">1</span><span class="p">)</span>
<span class="n">c</span> <span class="o">:=</span> <span class="n">factory</span><span class="p">(</span><span class="n">ReferenceCounter</span><span class="p">{</span>
<span class="n">count</span><span class="o">:</span> <span class="nb">new</span><span class="p">(</span><span class="kt">uint32</span><span class="p">),</span>
<span class="n">destination</span><span class="o">:</span> <span class="n">p</span><span class="o">.</span><span class="n">pool</span><span class="p">,</span>
<span class="n">released</span><span class="o">:</span> <span class="o">&</span><span class="n">p</span><span class="o">.</span><span class="n">returned</span><span class="p">,</span>
<span class="n">reset</span><span class="o">:</span> <span class="n">reset</span><span class="p">,</span>
<span class="n">id</span><span class="o">:</span> <span class="n">p</span><span class="o">.</span><span class="n">allocated</span><span class="p">,</span>
<span class="p">})</span>
<span class="k">return</span> <span class="n">c</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">p</span>
<span class="p">}</span>
<span class="c">// Method to get new object</span>
<span class="k">func</span> <span class="p">(</span><span class="n">p</span> <span class="o">*</span><span class="n">referenceCountedPool</span><span class="p">)</span> <span class="n">Get</span><span class="p">()</span> <span class="n">ReferenceCountable</span> <span class="p">{</span>
<span class="n">c</span> <span class="o">:=</span> <span class="n">p</span><span class="o">.</span><span class="n">pool</span><span class="o">.</span><span class="n">Get</span><span class="p">()</span><span class="o">.</span><span class="p">(</span><span class="n">ReferenceCountable</span><span class="p">)</span>
<span class="n">c</span><span class="o">.</span><span class="n">SetInstance</span><span class="p">(</span><span class="n">c</span><span class="p">)</span>
<span class="n">atomic</span><span class="o">.</span><span class="n">AddUint32</span><span class="p">(</span><span class="o">&</span><span class="n">p</span><span class="o">.</span><span class="n">referenced</span><span class="p">,</span> <span class="m">1</span><span class="p">)</span>
<span class="n">c</span><span class="o">.</span><span class="n">IncrementReferenceCount</span><span class="p">()</span>
<span class="k">return</span> <span class="n">c</span>
<span class="p">}</span>
<span class="c">// Method to return reference counted pool stats</span>
<span class="k">func</span> <span class="p">(</span><span class="n">p</span> <span class="o">*</span><span class="n">referenceCountedPool</span><span class="p">)</span> <span class="n">Stats</span><span class="p">()</span> <span class="k">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="k">interface</span><span class="p">{}</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="k">interface</span><span class="p">{}{</span><span class="s">"allocated"</span><span class="o">:</span> <span class="n">p</span><span class="o">.</span><span class="n">allocated</span><span class="p">,</span> <span class="s">"referenced"</span><span class="o">:</span> <span class="n">p</span><span class="o">.</span><span class="n">referenced</span><span class="p">,</span> <span class="s">"returned"</span><span class="o">:</span> <span class="n">p</span><span class="o">.</span><span class="n">returned</span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<ul>
<li><code class="language-plaintext highlighter-rouge">ReferenceCountedPool</code> expects a factory method that returns <code class="language-plaintext highlighter-rouge">ReferenceCoutable</code> instance i.e. an object implementing <code class="language-plaintext highlighter-rouge">ReferenceCountable</code>. Here in this example we will be embedding <code class="language-plaintext highlighter-rouge">ReferenceCounter</code> which will suffice this condition.</li>
</ul>
<hr />
<p>Now start using the ReferenceCoutedPool as following:</p>
<p>Consider following struct <code class="language-plaintext highlighter-rouge">Event</code></p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// Struct representing an event</span>
<span class="k">type</span> <span class="n">Event</span> <span class="k">struct</span> <span class="p">{</span>
<span class="n">pool</span><span class="o">.</span><span class="n">ReferenceCounter</span> <span class="s">`sql:"-"`</span>
<span class="n">Name</span> <span class="kt">string</span> <span class="s">`json:"name"`</span>
<span class="n">Log</span> <span class="kt">string</span> <span class="s">`json:"log"`</span>
<span class="n">Timestamp</span> <span class="n">time</span><span class="o">.</span><span class="n">Time</span> <span class="s">`json:"timestamp"`</span>
<span class="p">}</span>
<span class="c">// Method to reset event</span>
<span class="k">func</span> <span class="p">(</span><span class="n">e</span> <span class="o">*</span><span class="n">Event</span><span class="p">)</span> <span class="n">Reset</span><span class="p">()</span> <span class="p">{</span>
<span class="n">e</span><span class="o">.</span><span class="n">Name</span> <span class="o">=</span> <span class="s">""</span>
<span class="n">e</span><span class="o">.</span><span class="n">Log</span> <span class="o">=</span> <span class="s">""</span>
<span class="n">e</span><span class="o">.</span><span class="n">Timestamp</span> <span class="o">=</span> <span class="m">0</span>
<span class="p">}</span>
<span class="c">// Method to reset Event</span>
<span class="c">// Used by reference countable pool</span>
<span class="k">func</span> <span class="n">ResetEvent</span><span class="p">(</span><span class="n">i</span> <span class="k">interface</span><span class="p">{})</span> <span class="kt">error</span> <span class="p">{</span>
<span class="n">obj</span><span class="p">,</span> <span class="n">ok</span> <span class="o">:=</span> <span class="n">i</span><span class="o">.</span><span class="p">(</span><span class="o">*</span><span class="n">Event</span><span class="p">)</span>
<span class="k">if</span> <span class="o">!</span><span class="n">ok</span> <span class="p">{</span>
<span class="n">errors</span><span class="o">.</span><span class="n">New</span><span class="p">(</span><span class="s">"illegal object sent to ResetEvent"</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">obj</span><span class="o">.</span><span class="n">Reset</span><span class="p">()</span>
<span class="k">return</span> <span class="no">nil</span>
<span class="p">}</span>
<span class="c">// Method to create new event</span>
<span class="k">func</span> <span class="n">NewEvent</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">log</span> <span class="kt">string</span><span class="p">,</span> <span class="n">timestamp</span> <span class="n">time</span><span class="o">.</span><span class="n">Time</span><span class="p">)</span> <span class="o">*</span><span class="n">Event</span> <span class="p">{</span>
<span class="n">e</span> <span class="o">:=</span> <span class="n">AcquireEvent</span><span class="p">()</span>
<span class="n">e</span><span class="o">.</span><span class="n">Name</span> <span class="o">=</span> <span class="n">name</span>
<span class="n">e</span><span class="o">.</span><span class="n">Log</span> <span class="o">=</span> <span class="n">log</span>
<span class="n">e</span><span class="o">.</span><span class="n">Timestamp</span> <span class="o">=</span> <span class="n">timestamp</span>
<span class="k">return</span> <span class="n">e</span>
<span class="p">}</span>
</code></pre></div></div>
<p>And finally the Event pool</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// Event pool</span>
<span class="k">var</span> <span class="n">eventPool</span> <span class="o">=</span> <span class="n">pool</span><span class="o">.</span><span class="n">NewReferenceCountedPool</span><span class="p">(</span>
<span class="k">func</span><span class="p">(</span><span class="n">counter</span> <span class="n">pool</span><span class="o">.</span><span class="n">ReferenceCounter</span><span class="p">)</span> <span class="n">pool</span><span class="o">.</span><span class="n">ReferenceCountable</span> <span class="p">{</span>
<span class="n">br</span> <span class="o">:=</span> <span class="nb">new</span><span class="p">(</span><span class="n">Event</span><span class="p">)</span>
<span class="n">br</span><span class="o">.</span><span class="n">ReferenceCounter</span> <span class="o">=</span> <span class="n">counter</span>
<span class="k">return</span> <span class="n">br</span>
<span class="p">},</span> <span class="n">ResetEvent</span><span class="p">)</span>
<span class="c">// Method to get new Event</span>
<span class="k">func</span> <span class="n">AcquireEvent</span><span class="p">()</span> <span class="o">*</span><span class="n">Event</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">eventPool</span><span class="o">.</span><span class="n">Get</span><span class="p">()</span><span class="o">.</span><span class="p">(</span><span class="o">*</span><span class="n">Event</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Basic usage:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">e</span> <span class="o">:=</span> <span class="n">models</span><span class="o">.</span><span class="n">NewEvent</span><span class="p">(</span><span class="s">"test"</span><span class="p">,</span> <span class="s">"this is a test log"</span><span class="p">,</span> <span class="n">time</span><span class="o">.</span><span class="n">Now</span><span class="p">())</span>
<span class="k">defer</span> <span class="n">e</span><span class="o">.</span><span class="n">DecrementReferenceCount</span><span class="p">()</span>
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>This entire method seems a bit of an overhead, and requires a lot more precision while coding. You miss Reference counting at one place and it could result in a lot of unwanted results. But this works like a charm (see the benchmarks) once you get it right.</p>
<p>Let me know your opinions, doubts, arguments here or at akshaymdeo[at]gmail.com. Happy coding \m/</p>
<h2 id="references">References</h2>
<ul>
<li>[1] https://golang.org/pkg/sync/#Pool</li>
<li>[2] https://github.com/valyala/fasthttp</li>
<li>[3] https://github.com/rs/zerolog</li>
<li>[4] http://www.hydrogen18.com/blog/reference-counted-pool-golang.html</li>
</ul>Akshay DeoWe @media.net write superfast backends with at max 30-40ms turn-around time from web-service. We continuously try to reduce money spent per request. This blog enlists a few of our findings.Golang: Returning errors with context2017-09-05T06:25:46+00:002017-09-05T06:25:46+00:00https://www.akshaydeo.com/blog/2017/09/05/Golang-return-errors-with-context<p>Usually in Golang, it is recommended that, while propagating errors outside, just return the error do not log it. <a href="https://dave.cheney.net/2015/11/05/lets-talk-about-logging">This</a> blog by Dave Cheney talks in length about how shall we handle logging and errors. But while returning/propagating errors, somtimes it becomes necessary to add the context of the error along with the actual error.</p>
<p><strong>Basic example is:</strong></p>
<p>Consider your web app is talking to another gRPC server for getting location information through a function called, <code class="language-plaintext highlighter-rouge">GetLocationFor(user)</code>. Now there could be long enough function call tree that lands us onto this function. So if something goes wrong with gRPC connection, and if we return the error as is, technically we have lost the context.</p>
<figure class="highlight"><pre><code class="language-golang" data-lang="golang"><span class="k">func</span> <span class="n">GetLocationFor</span><span class="p">(</span><span class="n">u</span> <span class="o">*</span><span class="n">User</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="n">Location</span><span class="p">,</span><span class="kt">error</span><span class="p">){</span>
<span class="n">respMsg</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">grpcClient</span><span class="o">.</span><span class="n">GetLocationFor</span><span class="p">(</span><span class="n">u</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span><span class="p">{</span>
<span class="c">// here we directly send the error</span>
<span class="k">return</span> <span class="no">nil</span><span class="p">,</span> <span class="n">err</span>
<span class="p">}</span>
<span class="c">// process the respMsg and move on</span>
<span class="p">}</span></code></pre></figure>
<p>So there is one better way to just return the error (without logging it) and keeping the context:</p>
<p>Consider following supporting method for creating error objects</p>
<figure class="highlight"><pre><code class="language-golang" data-lang="golang"><span class="k">func</span> <span class="n">New</span><span class="p">(</span><span class="n">args</span> <span class="o">...</span><span class="k">interface</span><span class="p">{})</span> <span class="kt">error</span> <span class="p">{</span>
<span class="k">var</span> <span class="n">err</span> <span class="kt">error</span>
<span class="k">var</span> <span class="n">rawData</span> <span class="p">[]</span><span class="k">interface</span><span class="p">{}</span>
<span class="k">for</span> <span class="err">\</span><span class="n">_</span><span class="p">,</span> <span class="n">arg</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">args</span> <span class="p">{</span>
<span class="k">switch</span> <span class="n">arg</span><span class="o">.</span><span class="p">(</span><span class="k">type</span><span class="p">)</span> <span class="p">{</span>
<span class="k">case</span> <span class="kt">error</span><span class="o">:</span>
<span class="n">err</span> <span class="o">=</span> <span class="n">arg</span><span class="o">.</span><span class="p">(</span><span class="kt">error</span><span class="p">)</span>
<span class="n">log</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="s">"error"</span><span class="p">,</span> <span class="n">err</span><span class="p">)</span>
<span class="k">continue</span>
<span class="k">default</span><span class="o">:</span>
<span class="n">rawData</span> <span class="o">=</span> <span class="nb">append</span><span class="p">(</span><span class="n">rawData</span><span class="p">,</span> <span class="n">arg</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">==</span> <span class="no">nil</span> <span class="p">{</span>
<span class="n">err</span> <span class="o">=</span> <span class="n">errors</span><span class="o">.</span><span class="n">New</span><span class="p">(</span><span class="n">fmt</span><span class="o">.</span><span class="n">Sprintf</span><span class="p">(</span><span class="s">"%v"</span><span class="p">,</span> <span class="n">rawData</span><span class="p">))</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">errors</span><span class="o">.</span><span class="n">New</span><span class="p">(</span><span class="n">fmt</span><span class="o">.</span><span class="n">Sprintf</span><span class="p">(</span><span class="s">"%v [error => %s]"</span><span class="p">,</span> <span class="n">rawData</span><span class="p">,</span> <span class="n">err</span><span class="o">.</span><span class="n">Error</span><span class="p">()))</span>
<span class="p">}</span></code></pre></figure>
<p>And use it as</p>
<figure class="highlight"><pre><code class="language-golang" data-lang="golang"><span class="k">import</span> <span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">akshaydeo</span><span class="o">/</span><span class="n">errors</span>
<span class="k">func</span> <span class="n">GetLocationFor</span><span class="p">(</span><span class="n">u</span> <span class="o">*</span><span class="n">User</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="n">Location</span><span class="p">,</span><span class="kt">error</span><span class="p">){</span>
<span class="n">respMsg</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">grpcClient</span><span class="o">.</span><span class="n">GetLocationFor</span><span class="p">(</span><span class="n">u</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span><span class="p">{</span>
<span class="c">// here we directly send the error</span>
<span class="k">return</span> <span class="no">nil</span><span class="p">,</span> <span class="n">errors</span><span class="o">.</span><span class="n">New</span><span class="p">(</span><span class="s">"while getting location from grpc client in GetLocationFor"</span><span class="p">,</span> <span class="n">err</span><span class="p">)</span>
<span class="p">}</span>
<span class="c">// process the respMsg and move on</span>
<span class="p">}</span></code></pre></figure>
<p>After this, whenver you log the error it will be something like <code class="language-plaintext highlighter-rouge">while getting location from grpc client in GetLocatioFor [error => <original_error_message_from_grpc_client>]</code></p>
<p>This keeps the context of the error very specific and makes it easier to pinpoint the exact issue.</p>
<p>Happy coding \m/</p>Akshay DeoUsually in Golang, it is recommended that, while propagating errors outside, just return the error do not log it. This blog by Dave Cheney talks in length about how shall we handle logging and errors. But while returning/propagating errors, somtimes it becomes necessary to add the context of the error along with the actual error.Shipping your Android SDK anytime on live devices2016-12-21T06:25:46+00:002016-12-21T06:25:46+00:00https://www.akshaydeo.com/blog/2016/12/21/shipping-your-android-sdk-anytime-on-live-devices<blockquote>
<p>As per the new Play store guidelines, this method is categorized as an illegal way of executing any functionality on user’s device. I personally won’t recommend this method anymore.</p>
</blockquote>
<p>If your product depends upon a mobile SDK, then you must be knowing the real pain of shipping the latest version of your SDK onto the live devices through host apps. There is a way to tackle this issue with an interesting approach.</p>
<p>We are going to use dynamic class loading using classloaders provided by Android system.</p>
<h3 id="class-loader">Class loader</h3>
<p>The Java classloader is a part of the Java Runtime Environment that dynamically loads Java classes into the Java Virtual Machine. Usually, classes are only loaded on demand. The Java runtime system does not need to know about files and file systems because of classloaders.[2]</p>
<p>We are gonna use a special class loader from Android system called <a href="https://developer.android.com/reference/dalvik/system/PathClassLoader.html">PathClassLoader</a>, which provides a simple ClassLoader implementation that operates on a list of files and directories in the local file system, but does not attempt to load classes from the network. Android uses this class for its system class loader and for its application class loader(s).</p>
<h3 id="chaining-the-classloaders">Chaining the ClassLoaders</h3>
<p><img src="/public/images/classloading.jpg" alt="" />[3]</p>
<p>So we can create a hierarchy of classloaders that can share the definition of the classes without any duplication. So in this case, classloader A has already loaded class A, so <code class="language-plaintext highlighter-rouge">loaderB.loadClass('A')</code> will delegate the request to <code class="language-plaintext highlighter-rouge">loaderA</code> instead of reloading it. This feature comes handy for our purpose.</p>
<h3 id="application-lifecycle-on-android-os">Application lifecycle on Android OS</h3>
<p><img src="/public/images/app_launch.jpg" alt="" />[4]</p>
<p>When a user clicks on the app icon, a new application thread is started which contains a new instance of the dalvik vm. This dalvik vm has a classloader that loads the dex file (APK file) and kicks off the lifecycle of the app.</p>
<h3 id="what-if-we-provide-one-more-dex-to-load-at-runtime">What if we provide one more dex to load at runtime?</h3>
<p>Consider the AAR file that we ship, contains one more DEX file which is loaded by the AAR at the time of initialization. This will allow us to change the functionality without having to force update the AAR on the devices.</p>
<h3 id="structure-of-the-project">Structure of the project</h3>
<p><img src="/public/images/project_structure_for_injection.png" alt="" /></p>
<p><strong>app</strong> : Host app where you will integrate the SDK</p>
<figure class="highlight"><pre><code class="language-groovy" data-lang="groovy"><span class="n">dependencies</span> <span class="o">{</span>
<span class="o">...</span>
<span class="n">compile</span> <span class="nf">project</span><span class="o">(</span><span class="s2">":lib"</span><span class="o">)</span>
<span class="o">...</span>
<span class="o">}</span></code></pre></figure>
<p><strong>lib</strong> : Our static SDK code that ships with the app. This has the secret sauce of loading the functionality runtime.</p>
<p><strong>dynamiclib</strong> : SDK that does the implementation of the dynamic functionality of the SDK.</p>
<p>Let’s divide our tasks and decode one by one.</p>
<h4 id="code-architecture">Code architecture</h4>
<p>To bring in the dynamic features, I am going with an interface approach. So our dynamic lib and lib will share one common interface.</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">IMethods</span> <span class="o">{</span>
<span class="nc">String</span> <span class="nf">getVersion</span><span class="o">();</span>
<span class="kt">void</span> <span class="nf">log</span><span class="o">(</span><span class="nc">String</span> <span class="n">message</span><span class="o">);</span>
<span class="o">}</span></code></pre></figure>
<blockquote>
<p>Make sure that the package name for this interface will be the exactly same in both <code class="language-plaintext highlighter-rouge">lib</code> and <code class="language-plaintext highlighter-rouge">dynamiclib</code> module.</p>
</blockquote>
<p>This <code class="language-plaintext highlighter-rouge">IMethods</code> is implemented by a class called <code class="language-plaintext highlighter-rouge">MethodsImpl</code> in dynamic lib.</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MethodsImpl</span> <span class="kd">implements</span> <span class="nc">IMethods</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="no">TAG</span> <span class="o">=</span> <span class="s">"##MethodsImpl##"</span><span class="o">;</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="nc">String</span> <span class="nf">getVersion</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="s">"0.1.1"</span><span class="o">;</span>
<span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">log</span><span class="o">(</span><span class="nc">String</span> <span class="n">message</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">Log</span><span class="o">.</span><span class="na">d</span><span class="o">(</span><span class="no">TAG</span><span class="o">,</span> <span class="n">message</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<p>Now build this dynamiclib to an APK (as we need a dex file), and host it. I use SimpleHttpServer.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">python <span class="nt">-m</span> SimpleHTTPServer 8000</code></pre></figure>
<p>This makes dynamic lib available on a link.</p>
<p>Let’s build the last leg of the solution, downloading the APK and loading the classes using a PathClassLoader.</p>
<h5 id="downloading-the-apk">Downloading the APK</h5>
<p>Use any way to download the APK, I have used an AsyncTask</p>
<h5 id="store-the-downloaded-apk-on-internal-storage-sandboxed-for-our-package">Store the downloaded APK on internal storage sandboxed for our package</h5>
<p><code class="language-plaintext highlighter-rouge">context.getFilesDir()</code> gives the path to the internal sandboxed storage for the package. Store the downloaded APK in this folder.</p>
<h5 id="load-the-downloaded-apk-and-cast-it-to-the-imethods">Load the downloaded APK and cast it to the IMethods</h5>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="c1">// INTERNAL_DEX_PATH = context.getFilesDir() + FILE_NAME this is path to the file we have downloaded</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">loadSdk</span><span class="o">(</span><span class="nc">Context</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">PathClassLoader</span> <span class="n">pathClassLoader</span> <span class="o">=</span>
<span class="k">new</span> <span class="nf">PathClassLoader</span><span class="o">(</span><span class="no">INTERNAL_DEX_PATH</span><span class="o">,</span> <span class="n">context</span><span class="o">.</span><span class="na">getClassLoader</span><span class="o">());</span>
<span class="k">try</span> <span class="o">{</span>
<span class="nc">Log</span><span class="o">.</span><span class="na">d</span><span class="o">(</span><span class="no">TAG</span><span class="o">,</span> <span class="s">"loading sdk class"</span><span class="o">);</span>
<span class="c1">// This step load the implementation of IMethods dynamically</span>
<span class="nc">Class</span> <span class="n">sdkClass</span> <span class="o">=</span> <span class="n">pathClassLoader</span><span class="o">.</span><span class="na">loadClass</span><span class="o">(</span><span class="s">"net.media.dynamiclib.MethodsImpl"</span><span class="o">);</span>
<span class="c1">// This step creates the new instance of MethodsImpl and casts it to IMethods</span>
<span class="nc">IMethods</span> <span class="n">methods</span> <span class="o">=</span> <span class="o">(</span><span class="nc">IMethods</span><span class="o">)</span> <span class="n">sdkClass</span><span class="o">.</span><span class="na">newInstance</span><span class="o">();</span>
<span class="c1">// This should log this message on logcat</span>
<span class="n">methods</span><span class="o">.</span><span class="na">log</span><span class="o">(</span><span class="s">"testing this"</span><span class="o">);</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<p>Logs when you run the app is</p>
<figure class="highlight"><pre><code class="language-log" data-lang="log">12-22 17:16:29.036 10926-10926/net.media.injector D/## Injector ##: load()
12-22 17:16:29.131 10926-10926/net.media.injector W/gralloc_ranchu: Gralloc pipe failed
[ 12-22 17:16:29.132 10926:10926 D/ ]
HostConnection::get() New Host Connection established 0xad9f6d80, tid 10926
12-22 17:16:29.174 10926-10954/net.media.injector D/NetworkSecurityConfig: No Network Security Config specified, using platform default
12-22 17:16:29.222 10926-10954/net.media.injector D/## Injector ##: File output stream is => /data/user/0/net.media.injector/files/lib.apk
12-22 17:16:29.523 10926-10953/net.media.injector I/OpenGLRenderer: Initialized EGL, version 1.4
12-22 17:16:29.526 10926-10953/net.media.injector D/OpenGLRenderer: Swap behavior 1
12-22 17:16:30.064 10926-10926/net.media.injector D/## Injector ##: downloaded
12-22 17:16:30.064 10926-10926/net.media.injector D/## Injector ##: /data/user/0/net.media.injector/files/impl.dex
12-22 17:16:30.064 10926-10926/net.media.injector D/## Injector ##: /data/user/0/net.media.injector/files/impl.aar
12-22 17:16:30.065 10926-10926/net.media.injector D/## Injector ##: /data/user/0/net.media.injector/files/lib.apk
12-22 17:16:30.170 10926-10926/net.media.injector D/## Injector ##: loading sdk class
12-22 17:16:30.171 10926-10926/net.media.injector D/##MethodsImpl##: testing this</code></pre></figure>
<blockquote>
<p>This approach will allow you to update the functionality with a strong interface defined between shipped code and dynamic part of the SDK. This is the case where host app keeps interacting with your SDK using a set of functions. If your SDK is, initialize once and forget, then you can keep entire logic in dynamic part of the SDK.</p>
</blockquote>
<h3 id="security">Security</h3>
<p>Security is going to be one of the biggest concern in this approach. There are some standard ways to validate the dex file like using MD5 hashes. Once you securely download the DEX file in the internal storage then other concerns are as same as they are for shipped SDK.</p>
<p><a href="https://github.com/akshaydeo/Injector">Github repo</a> for the source code.</p>
<p>Get in touch with me if you need any help or you find something wrong with this post. Happy coding :).</p>
<hr />
<p><br />
<strong>References</strong></p>
<p>[1] <a href="http://www.electronicsweekly.com/blogs/eyes-on-android/what-is/the-dalvik-virtual-machine-2011-10/">Android system fundamentals</a></p>
<p>[2] <a href="https://en.wikipedia.org/wiki/Java_Classloader">ClassLoaders Wiki</a></p>
<p>[3] [How do classloaders work]((https://myprogressivelearning.wordpress.com/2014/10/28/class-loading-in-java-java-classloader-what-and-how/)</p>
<p>[4] <a href="https://coltf.blogspot.in/p/android-os-processes-and-zygote.html">Android Application lifecycle</a></p>Akshay DeoAs per the new Play store guidelines, this method is categorized as an illegal way of executing any functionality on user’s device. I personally won’t recommend this method anymore.How to use packages specifically for Debug/Release builds in Android2016-01-05T07:09:06+00:002016-01-05T07:09:06+00:00https://www.akshaydeo.com/blog/2016/01/05/how-to-use-packages-specifically-for-debug-slash-release-builds-in-android<h2 id="preface">Preface</h2>
<p>Currently I am working on an Android app for one of the most interesting startups in Fintech. I have been really choosy about the packages that are getting shipped with this app, simply because it involves a lot of money related functionalties. During the development, I came across a requirement that debug builds should have instabug integrated for reporting UI issues easily. APK size matters a lot, so I wanted to achieve this without shipping Instabug SDK in production builds.</p>
<h2 id="how-to-do-this">How to do this?</h2>
<h3 id="gradle-file">Gradle file</h3>
<figure class="highlight"><pre><code class="language-groovy" data-lang="groovy"><span class="o">...</span>
<span class="n">dependencies</span> <span class="o">{</span>
<span class="o">...</span>
<span class="n">compile</span> <span class="n">appDependencies</span><span class="o">.</span><span class="na">rateUs</span>
<span class="n">compile</span> <span class="n">appDependencies</span><span class="o">.</span><span class="na">markdownJ</span>
<span class="nf">debugCompile</span><span class="o">(</span><span class="n">appDependencies</span><span class="o">.</span><span class="na">instabug</span><span class="o">)</span> <span class="o">{</span>
<span class="n">exclude</span> <span class="nl">group:</span> <span class="s1">'com.mcxiaoke.volley'</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">...</span> </code></pre></figure>
<p>I changed the way gradle compiles instabug dependency. Now it’s done during debug builds only. Release builds will not consider this dependency.</p>
<h3 id="how-to-use-this-conditional-dependency-in-code">How to use this conditional dependency in code?</h3>
<p><img src="/public/images/1.png" alt="" />
<img src="/public/images/2.png" alt="" /></p>
<p>Create debug and release folders inside the src folder of your app. Create the exact same package structure (same as main folder) in both of these folders. Create a class with overriding nature i.e. same name and methods. Now write add instabug initialization inside the debug flavour, while release flavour won’t have this bit. Update your main Application class to include the initialization of this newly created class.</p>
<p>Application class in main.</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">onCreate</span><span class="o">()</span> <span class="o">{</span>
<span class="kd">super</span><span class="o">.</span><span class="na">onCreate</span><span class="o">();</span>
<span class="o">...</span>
<span class="nc">SimplAppInitializer</span><span class="o">.</span><span class="na">init</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
<span class="o">...</span>
<span class="o">}</span></code></pre></figure>
<p>And you are done, now the instabug will be only compiled with the debug builds and not with your production builds.</p>
<p>Let me know if there are some issues/suggestions related to this post.
Happy coding \m/</p>Akshay DeoPreface