Holy on Dev

Posts Tagged ‘linux’

Most interesting links of July ’14

Posted by Jakub Holý on July 31, 2014

Recommended Readings

  • Video: The Unreasonable Effectiveness of Dynamic Typing for Practical Programs – a static-typing zealot turned friend of dynamic typing under the experience of real-world projects and problems shares thoughts about the limits of type systems (f.ex. both energy and torque are measured in N*m yet cannot be combined) and their cost: according to the Hanenberg’s experiment about static and dynamic typing => the time required to handle the time chacker > time to debug the errors that it would have caught. According to a review of issues at GitHub, only 2% of reported issues for JS, Clojure, Python, and Ruby are type errors and for a large, closed-source Python project type/name/attribute errors were 1%. “I have come to believe that tests are a much better investment [than static typing].” Rigorous type system/model => limited applicability (due to different needs) <=> modelling some things with types doesn’t cut it. “Are the costs of static typing offset by a few percent fewer defects? Is agility more important than reliability?” “Static types are anti-modular” – too a tight coupling. “Static type checking comes at the expense of complexity, brittleness and a tendency to monoliths.
    (Personally I miss static typing – but that is perhaps due to having relied on it for so long.)
  • ThoughtWorks Tech Radar July 2014 (pdf): f.ex. Ansible in Adapt, Masterless Chef/Puppet in Trial, Machine image as a build artifact: Trial, PostgreSQL for NoSQL: Trial, Adopt Dropwizard (Rest 4 Java), Go lang, Reactive Extensions across langs [JH: RxJava, RxJS, ..]; Asses Property-based (generative) testing, … . Other highlights: Mapbox (open-source mapping platform), OpenID Connect as a less complex and thus promising alternative to SAML/generic OAuth, Pacto/Pact for Consumer-Driven Contracts (contract => simulate consumers/stubb producers => test your REST clients against the contract so that the rest of tests can assume it is correct and use a stubbed client), Swagger for REST documentation.
  • The madness of layered architecture – a nice critique of over-designed “enterprise” apps, why that is a problem (SRP, cost of code, unclear where to do a change, ….), why it is different from the successful layered network stack of Ethernet/IP/TCP/… (because in an app, all layers are on the same level of abstraction); bottom line: do not add a layer unless you have a really good reason (hint: the advice of a consultant/speaker does not count as one)
  • Key Takeaway Points and Lessons Learned from QCon New York 2014 (viz @RiczWest) – “[..] deep insights into real-world architectures and state of the art software development practices, from a practioner’s perspective.” – architectures of Fb, Foursquare etc., continuous delivery, creating culture, real world functional programming, … .
  • Questioning the Lambda Architecture (J. Kreps of LinkedIn) – maintaining the same processing in two very different systems (one batch, one stream & real-time) is a maintenance nightmare => improve the RT/stream processing to handle re-processing and thus both (using e.g. Kafka to store the data and thus be able to re-play them)
  • Google: Checklist for mobile website improvement
  • Google Dataflow and the transition from batch to stream processing – G. Dataflow might not be a Hadoop killer due to requiring that the data are in the Google Cloud but the trend is clear, going away from batch processing to more stream-oriented processing with tools like Spark, Flume etc. that are faster thanks to using memory better and more flexible thanks to not being limited to the rigitd two-stage model of map-reduce. (Reportedly, Google – the one that made Map-Reduce popular – doesn’t use it anymore.)
  • OS X: Extract JDK to folder, without running installer

Society, economics, people

  • HBR: The Power of Meeting Your Employees’ Needs – people feel better, perform better, are more engaged and likely to stay longer (=> profitability) when 4 basic needs are met: physical [energy] renewal (=> give opportunity, encourage to take a nap or do whatever that helps), value – feeling of being valued by the company, ability to focus, purpose (i.e. serving something larger than ourselves). “What’s surprising about our survey’s results is how dramatically and positively getting these needs met is correlated with every variable that influences performance. It would be statistically significant if meeting a given need correlated with a rise of even one or two percentage points in a performance variable such as engagement, or retention. Instead, we found that meeting even one of the four core needs had a dramatic impact on every performance variable we studied. [..] when all four needs are met, the effect on engagement rises from 50% for one need, to 125%. Engagement, in turn, has been positively correlated with profitability. [..] employers with the most engaged employees were 22% more profitable than those with the least engaged employees.
    [..] those who were encouraged to take intermittent breaks reported they were 50% more engaged, more than twice as likely to stay with the company, and twice as healthy overall. Valuing and encouraging renewal requires no financial investment. What it does require is a willingness among leaders to test their longstanding assumption that that performance is best measured by the number of hours employees puts in – and the more continuous the better — rather than by the value they generate, however they choose to do their work.
  • The Pitchforks Are Coming… For Us Plutocrats – increasing inequality will eventually lead to the collapse of the sysem (at least so does teach the history). It is people – primarily the middle class – that are the source of the wealth of the society, they produce and also consume most. Thus it is necessary to support them …
  • Why the U.S. Corporate World Became ‘A Bull Market for Corruption’ – Enron, GM, Goldman Sachs, … – we hear more and more the names of large corporations in the context of negligence and misues of their customers and investors. It seems that leadership (in the lead by example sense) has died out as well as the feeling of responsibility when one wields power over her customers/investors/markets. Instead, we have the me-first and  money at any cost thinking. Organizations are designed to shield higher-ups from responsibility (meetings with no records…). High pay for short term gains, failure to punish high ranking people.
  • (US) This is what happened when I drove my Mercedes to pick up food stamps – the experience of life in poverty after dropping down from $125k to $25k/year in two months due to childbirth, real estate market crash, and loss of a job. “Using the coupons was even worse. The stares, the faux concern, the pity, the outrage — I hated it. [..] That’s the funny thing about being poor. Everyone has an opinion on it, and everyone feels entitled to share. [..] Poverty is a circumstance, not a value judgment. I still have to remind myself sometimes that I was my harshest critic. That the judgment of the disadvantaged comes not just from conservative politicians and Internet trolls. It came from me, even as I was living it.

Clojure Corner

  • Isomorphic Clojure[Script], part I – enjoying all the benefits of Single-Page Apps while avoiding their drawbacks (SEO, slower page load, accessibility etc.) – a SPA that can be pre-rendered by the server. Using Om/React, JDK8 with the Nashorn JS engine, core.async, Sente (bi-dirrectional HTTP/WS communication over core.async) and Clojure in the JVM, ClojureScript in Nashorn in the JVM, and ClojureScript in the browser. Example app: Omelette.
  • clj-crud: a relatively feature-complete example of a Clojure web (4/2014; GitHub) – using Component, Liberator (REST), Datascript + Quiescent (=> React.js), Enlive, Friend etc. including couple of unit-test and ui-test libraries
  • Conclujon: Acceptance testing tool (α), Clojure reimplementation of Concordion, a beautifully simple ADD tool
  • dynalint: human-friendly error messages during dev – Clojure typically provides little helpful and sometimes confusing error messages thrown from somewhere deep in the implementation, such as “Don’t know how to create ISeq from: java.lang.Long at clojure.lang.RT.seqFrom” while we want st. like “First argument to clojure.core/first must be seqable: 1 (instance of class java.lang.Long” – and that’s what Dynalint does. In the tradition of defensive programming, it adds checks and good error messages to Vars at runtime. You typically run it only during dev, triggering it from the REPL.
  • Grimoire (Reid McKenzie) – a more up-to-date replacement for ClojureDocs
  • Adam Bard’s Top Clojure Articles for beginners and intermediate Clojure devs – f.ex. Five Mistakes Clojure Newbies Make, Acceptable Error Handling in Clojure, Clojure Reducers for Mortals
  • J. Wilk: Isolating External Dependencies in Clojure – a nice overview of the options and their pros and cons – with-redefs, alter-var-root, Midje (using alter-var-root in a more controlled manner), higher-order-functions (#1!) etc.
  • philandstuff’s detailed notes from Euroclojure 2014


  • NixOS (via @bodil) – a new interesting “purely functional” Linux distribution – system configuration is fully declarative (think of Puppet/Chef) and it is always trivial to roll back, you can have multiple versions of a package, users can install non-global SW
  • InfluxDB – time series, metrics, and events DB that scales; contrary to Graphite it can store richer data than Graphite and its single value; additional highlights: authorization for individual data, roll-up/clean old data, https API. Written in Go.

Posted in General, Languages, Top links of month | Tagged: , , , , , , , , , | Comments Off on Most interesting links of July ’14

Running A Leiningen/Ring Webapp As A Daemon Via Upstart (Ubuntu)

Posted by Jakub Holý on July 27, 2013

Running a Java/Clojure app as a daemon on Linux used to be hard but is pretty simple with Ubuntu Upstart (docs). The short story:

  1. Create an all-in one uberjar via “lein with-profile production ring uberjar” (using the lein-ring plugin; a simple lein uberjar would suffice for an app with a main- method)
  2. Create an upstart <service name>.conf file in /etc/init/
  3. Run sudo start/stop/status <service name>

And of course it works with Puppet too.

Read the rest of this entry »

Posted in General | Tagged: , , | Comments Off on Running A Leiningen/Ring Webapp As A Daemon Via Upstart (Ubuntu)

Tool Tip: Byob – Screen With Text UI

Posted by Jakub Holý on October 17, 2012

Screen (man) is very useful for running terminal sessions on remote computers that enable the user to disconnect and re-connect. Byobu (man), formerly also called screen-profiles, is a wrapper script for screen that adds status lines with useful info to screen and provides text UI for configuring it (byobu-config).

Screen allows you to have multiple “windows” (terminal sessions) opened at the same time and jump between them (C-a n). You can also display multiple windows if you split your screen.

Few screen/byobu tips (C-a means Contol-a):

  • Configuration
    • Increase scroll history via “screen -h <num lines>” or “defscrollback <num lines>” in ~/.screenrc
  • Detach from screen: C-a d
  • Re-attach to screen: screen -rd
  • Split vertically: C-a |
    • Jump to the new split (or back): C-a tab
    • Create new window in the split (=> shell): C-a c
    • Remove all splits but the current one: C-a Q
  • Next window: C-a n
  • Pass “C-a” to the terminal running under screen: “C-a a”
  • Help “C-a ?”
  • Byobu: disable the Fn keys and only use the traditional screen commands: run byobu-config and select to use only them
    • To access Bybou menu (config) type, instead of F9, C-a @

Posted in Tools | Tagged: | Comments Off on Tool Tip: Byob – Screen With Text UI

Using Java as Native Linux Apps – Calling C, Daemonization, Packaging, CLI (Brian McCallister)

Posted by Jakub Holý on September 25, 2012

This is a summary of the excellent JavaZone 2012 talk Going Native (vimeo) by Brian McCallister. Content: Using native libraries in Java and packaging them with Java apps, daemonization, trully executable JARs, powerful CLI, creating manpages, packaging natively as deb/rpm.

1. Using Native Libs in Java

Calling Native Libs

Calling native libraries such as C ones was hard and ugly with JNI but is very simple and nice with JNA (GPL) and JNR (Apache/LGPL)
Read the rest of this entry »

Posted in Languages | Tagged: , , , , , | Comments Off on Using Java as Native Linux Apps – Calling C, Daemonization, Packaging, CLI (Brian McCallister)

AWK: Extract Logs for the Given Date(s) from a Log File

Posted by Jakub Holý on December 18, 2011

If your log file has entries like these:

2011-12-10T22:00:27.996+0000 [http-8080-1] INFO  my.package.MyClass Hello, I'm alive!
2011-12-11T17:05:46.811+0000 [http-8080-15] ERROR my.package.MyClass  - Error caught in DispatcherServlet
        at my.package.MyServiceClass(MyServiceClass.java:36)
2011-12-11T17:06:10.120+0000 [http-8080-14] DEBUG my.package.MyClass Whoo, that has been a long day!

Then you can use the following bash script snippet to extract logs only for a particular day or consecutive days, including everything – even lines not starting with the date such as stacktraces – between the first log of the date up to the first log of a subsequent date (default: yesterday):

if [ -z "$LOGDAY" ]; then LOGDAY=$(date +%F -d "-1 days"); fi
if [ -z "$AFTERLOGDAY" ]; then AFTERLOGDAY=$(date +%F -d "$LOGDAY +1 days"); fi
echo "Extracting logs in the range (>= $LOGDAY && < $AFTERLOGDAY) into $LOGFILE ..." awk "/^$LOGDAY/,/^$AFTERLOGDAY/ {if(!/^$AFTERLOGDAY/) print}" $LOGFILE_ORIG > $LOGFILE

This date format works on Linux. Date is very flexible and can provide dates in any format, not only yyyy-mm-dd. You may also want to read more about Awk ranges and other tips.

You would run it in one of the following ways:

$ ./analysis.sh /path/to/logfile.log
$ LOGDAY=2011-12-12 AFTERLOGDAY=2011-12-17 ./analysis.sh /path/to/logfile.log

Posted in General, Tools | Tagged: , , | Comments Off on AWK: Extract Logs for the Given Date(s) from a Log File

SSH magic: Authorize only once for multiple ssh/scp invocations

Posted by Jakub Holý on September 10, 2010

OpenSSH has a nice feature that makes it possible to open one “master connection”, which can be shared by multiple subsequent ssh/scp/sftp “slave connections”. The advantage is that you need to supply password only when opening the master connection and thus you can easily perform a sequence of remote commands without constant re-authentication. Let’s see how to do it in such a way that it can be used in a script.
Read the rest of this entry »

Posted in General | Tagged: , , , | Comments Off on SSH magic: Authorize only once for multiple ssh/scp invocations

An encrypted backup of a disk/partition to a Samba share with Clonezilla

Posted by Jakub Holý on August 10, 2010

You will learn how to customize Clonezilla Live (v. 1.2.5-24) for an easy backup of a partition (or a disk) to an encrypted file stored on a remote Samba server and how to test the backup by restoring it to a VMware virtual machine. We will few scripts to simplify the task, including a custom Clonezilla startup script to mount a TrueCrypt volume on a Samba share.

Content: What is Clonezilla? | Preparing Clonezilla for a custom backup | Backup | Encryption of the backup | Restoration and testing of the backup | The complete backup – encrypt – test cycle | Summary

PS: If you are scared by the length of this post then read only “The complete backup – encrypt – test cycle” 🙂

Update 2010-09-23: Added “The complete backup – encrypt – test cycle”, little reorganization.

What is Clonezilla?

Clonezilla is a live Linux distribution containing tools for performing backup and restoration of disks and partitions. It is basically a collection of various open-source tools such as partimage and gzip and custom scripts that “glue” them together to create a single backup tool driven by a wizard-like user interface. You install it to a CD or USB flash disk, boot from that medium, answer few questions and a backup or a restoration may start.

Read the rest of this entry »

Posted in Tools | Tagged: , , | Comments Off on An encrypted backup of a disk/partition to a Samba share with Clonezilla