So we start with the data flow diagram, or what happened to the data before we saw it on a page. <\/p>\n\n\n\n
What happens to your data?<\/h1>\n\n\n\n
As a distributed systems engineer, I deal with a chain of services and need to figure out who does what. Systems are big and developed by several teams or even departments, so finding the root of a seemingly simple problem can be pretty complex. <\/p>\n\n\n\n
Let\u2019s imagine our client has a balance of $100001 and wants to know where the extra dollar came from. Opening the page with the balance data may make only one backend request for debit and credit numbers \u2013 balance is just the difference between debit and credit numbers. But the devil is always in details. God knows how many services work together to get billing data to that page. And who knows how many services update debit and credit for the balance data?<\/p>\n\n\n\n
To understand it, we can draw a data flow diagram where squares are data processes<\/strong>. Arrows represent the transition of data (and actual data), and squares with numbers are data storage.<\/p>\n\n\n\n You can choose whatever type of diagram you like or make your own rules. You aim to understand what is happening and do what works best for you. But if you share your diagrams, make sure your colleagues understand your style and can easily read it without misinterpreting it.<\/p>\n\n\n\n I prefer a different kind of diagram where squares are actual data and arrows are a process of data.<\/p>\n\n\n\n This diagram may help you localize some data processes in particular services of your system.<\/p>\n\n\n\n We haven\u2019t found our answer yet, but now we understand which data processes our system performs.<\/p>\n\n\n\n The next step is understanding which services or groups of services are responsible for a particular data process. Architecture diagrams can help you with that:<\/p>\n\n\n\n In the current example, I made it as simple as possible: squares are services, arrows are interactions between services, and I added extra marks for data processes. This diagram is very simple and suits a lot of cases, and a big bonus is that everybody understands it intuitively.<\/p>\n\n\n\n Drawing an architecture diagram helps us:<\/strong><\/p>\n\n\n\n At that point checking data in storage showed everything was fine. The balancing service sends correct numbers for debits and credits, but the client still got an incorrect balance. Was something wrong with our API proxy? Let\u2019s dig deeper.<\/p>\n\n\n\n Based on the previous investigation of data flow, we now draw how data goes through the component of API proxy services. <\/p>\n\n\n\n According to the schema, no data transformation happens in the service. All the components add new information and work as proxies between a client and backends. So if a client gets the wrong numbers for debit and credit right after our proxy, something must be wrong with the data representation.<\/p>\n\n\n\n This means we have to double-check responses from balance services and what we have after the API proxy. The difference between debit and credit is 100000, and the balance service returns precisely what we need:<\/p>\n\n\n\n Response from API proxy is.<\/p>\n\n\n\n Responses are different; if we look at the numbers, the API proxy has the wrong debit. After our investigation, we know there is an API proxy issue. <\/p>\n\n\n\n Yet, we also know that it is a NodeJs service. Could our numbers be too big for JS? We have a Long data type which has Long.MAX_VALUE= 9,223,372,036,854,775,807, and JS has Number.MAX_SAFE_INTEGER = 9,007,199,254,740,991.<\/p>\n\n\n\n A debit number is bigger than the Max Integer Js number for 100000, which causes overflow, and as a result, we have the wrong number for debit and the wrong balance, which is bigger for 1$ = 100001.<\/p>\n\n\n\n 9007199254840991 \u2013 9007199254740991 = 100000<\/p>\n\n\n\n 9007199254840992 \u2013 9007199254740991 = 100001<\/p>\n\n\n\n So we sketched out the problem, found the issue, and we could get to work fixing it.<\/p>\n\n\n\n This drawing approach isn\u2019t always necessary, and some would argue that it can be a waste of time, but it helps if:<\/p>\n\n\n\n![\"\"](\"https:\/\/shiftmag.dev\/wp-content\/uploads\/2023\/05\/data_flow-1024x516.png?x54910\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/shiftmag.dev\/wp-content\/uploads\/2023\/05\/data_flow_hand-1024x287.png?x54910\")
<\/figure>\n\n\n\nFrom data to services<\/h1>\n\n\n\n
![\"\"](\"https:\/\/shiftmag.dev\/wp-content\/uploads\/2023\/05\/service_architecture-1024x812.png?x54910\")
<\/figure>\n\n\n\n\n
Deep dive into the components<\/h1>\n\n\n\n
![\"\"](\"https:\/\/shiftmag.dev\/wp-content\/uploads\/2023\/05\/api_proxy-1024x515.png?x54910\")
<\/figure>\n\n\n\n{ \n\n\u202f\u202f\u202f\u202fdebit:\u202f9007199254840991, \n\n\u202f\u202f\u202f\u202fcredit:\u202f9007199254740991 \n\n} <\/code><\/pre>\n\n\n\n{ \n\n\u202f\u202f\u202f\u202fdebit:\u202f9007199254840992, \n\n\u202f\u202f\u202f\u202fcredit:\u202f9007199254740991 \n\n} <\/code><\/pre>\n\n\n\nWhen drawing comes in handy<\/strong><\/h2>\n\n\n\n