Most of our infrastructure and client facing services are in us-east-1 and we have lots of users connecting from different parts of the world including India. Of course there was a significant latency involved when users connect to US from other part of the world. And we wanted to test that, if a user from India connects to Mumbai region(faster handshake) and then that region uses VPC peering to us-east-1 to talk to other services. Here’s how it went:
About new Hadoop cluster we set up, the phoenix version bundled with HDP distribution(4.7) had some bugs which would make it impossible to use to run BI queries. There was no way provided by HDP to upgrade phoenix as we were using the latest version. Looking around on the internet, I found that manually we can replace the related jars and bins to have a new version in place. Continue reading Upgrading Apache Phoenix in HDP Cluster
Recently I was revisiting concepts of TCP protocol and that reminded me that there was also a thing called TCP Fast Open. Digging further on the same revealed a lot. We will briefly discuss how this enhancement works. What are the limitations. And later we will do the hands on and see how the TCP Fast Open drastically reduces the load time. Continue reading TCP Fast Open: In Action with Python
Journey to websocket was pretty long. I started with an idea to make an app which can play music in sync across the devices during college period. No wonder I couldn’t get thru it. Later this year I stumbled upon this new thing called WebSockets and they were intriguing. I thought I could finish that app with websockets (and I did, with partial success). Spinned of another app out of it. And websockets were on a roll. It was time I digged further in and ended up writing a websocket server. (GitHub link at the bottom) Continue reading Writing Simple WebSocket Server in Python: PyWSocket
So the CDH Cluster was replaced by HDP Cluster and everything was going smooth for time being. Until the time when I started getting a dead RegionServer. Frequently. So a deep dive was needed to dig out what indeed was happening. And it turned out to be a long dive. Continue reading HBase YouAreDeadException: Dead RegionServer due to GC Pause
You would have wondered if it’s possible to reconfigure the PL part without any interruption while PS is running Linux. Well, it’s possible and as simple as,
The PYNQ Linux is a fun, easy and maker-friendly Ubuntu 15.04 rootfs. It comes bundled with the PYNQ-Z1 board, and the official documentations doesn’t even utter a word on how to build or port this image on any other Zynq. Maybe because it’s too obvious how to do so.
As a part of migration from CDH cluster to HDP cluster, we also had to migrate OpenTSDB which was running on CDH cluster. There are many methods to copy/transfer data between clusters and what we used here was ExportSnapshot.
If you’re an FPGA fan or someone who’s got PYNQ board for fun, you might be having a hard time making it run Vivado SDK projects. That’s because, the PYNQ-Z1, the cheap Zynq-7020 board doesn’t have any popular DDR ram on board. You need to configure it by hand, however, tcl is at your rescue.
So you are setting up HBase! Congratulations! When it comes to tuning HBase there are so many things you can do. And most of the things will be dependent upon type of data you will be storing and it’s access patterns. So I will be saying this a lot: ‘value of this parameter depends upon your workload’. Here I will try to enlist some of the variables that you can tweak while tuning hbase. This list is not at all exhaustive. Continue reading Stuff You Can Do While Tuning HBase