Weird rocketry » dev

OpenWRT Chaos Calmer 15.05 on Dlink DIR-320 with CDMA modem

bananos — Thu, 07 Apr 2016 11:53:19 +0000

The goal of this article is to install latest OpenWRT firmware on a very popular DIR-320 Wi-Fi router (BRCM47XX based). The reason why you might do this is to be able to connect modern hardware(modems) to this pretty old legacy device.

Official firmware 3G modem support list is pretty limited and uses old kernel. On the other hand, there is an alternative DD-WRT firwmare with linux kernel 2.6.x which is pretty old also. Therefore, the only viable option is installing OpenWRT which ships the latest linux kernels 3.18.xx.

As a bonus sub-topic we’ll consider setting up an external CDMA(EVDO) modem and using it to access internet.

In order to proceed and experiment by yourself the following is necessary:

Dlink DIR-320 router (rev. A1/A2 only!, be careful as there are B1/B2/NRU versions which are completely different hardware!)
RJ45 CAT5 Patch cable
USB to Serial adapter (I’ve been able to successfully use cheap PL2303TA based USB TTL to RS232)
USB flash card or hard drive
USB 3G modem (in my case it’s Sierra AirCard 250U)
USB Hub (external power source is highly recommended)
x86_64 Linux host system (i’m using Ubuntu 15.10 x86_64)
a lot of patience :)

Install OpenWRT using Image builder for brcm47xx/legacy

On your host linux system, download and unpack OpenWRT image builder packages:

bananos@ubby:$ wget "https://downloads.openwrt.org/chaos_calmer/15.05.1/brcm47xx/legacy/OpenWrt-ImageBuilder-15.05.1-brcm47xx-legacy.Linux-x86_64.tar.bz2"
bananos@ubby:$ tar -xvvf "OpenWrt-ImageBuilder-15.05.1-brcm47xx-legacy.Linux-x86_64.tar.bz2"

After that, we’re going to assemble our custom firmware out of pre-built packages. Image builder was created specifically to overcome the slowness and complexity of compiling OpenWRT from sources.

bananos@ubby:$ sudo apt-get install -y subversion build-essential libncurses5-dev zlib1g-dev gawk git ccache gettext libssl-dev xsltproc
bananos@ubby:$ cd "OpenWrt-ImageBuilder-15.05.1-brcm47xx-legacy.Linux-x86_64"
bananos@ubby:$ make image PROFILE="Broadcom-b43" PACKAGES="base-files libc libgcc busybox kmod-usb2 kmod-usb-ohci kmod-usb-uhci kmod-usb-storage kmod-fs-ext4 block-mount kmod-scsi-core -dropbear mtd uci -opkg netifd fstools swconfig wpad-mini nvram -otrx kmod-leds-gpio kmod-gpio-button-hotplug kmod-ledtrig-default-on kmod-ledtrig-timer kmod-ledtrig-netdev dnsmasq iptables -ip6tables ppp ppp-mod-pppoe kmod-nf-nathelper firewall odhcpd odhcp6c"

make[1]: Entering directory '/home/bananos/Projects/openwrt_dir320/image_builder/OpenWrt-ImageBuilder-15.05.1-brcm47xx-legacy.Linux-x86_64'
echo 'Building images for brcm47xx - Broadcom SoC, all Ethernet, BCM43xx WiFi (b43, default)'
Building images for brcm47xx - Broadcom SoC, all Ethernet, BCM43xx WiFi (b43, default)
echo 'Packages: base-files block-mount busybox dnsmasq firewall fstools iptables kernel kmod-b43 kmod-b43legacy kmod-fs-ext4 kmod-gpio-button-hotplug kmod-leds-gpio kmod-ledtrig-default-on kmod-ledtrig-netdev kmod-ledtrig-timer kmod-nf-nathelper kmod-scsi-core kmod-usb-ohci kmod-usb-storage kmod-usb-uhci kmod-usb2 libc libgcc mtd netifd nvram odhcp6c odhcpd ppp ppp-mod-pppoe swconfig uci wpad-mini'
Packages: base-files block-mount busybox dnsmasq firewall fstools iptables kernel kmod-b43 kmod-b43legacy kmod-fs-ext4 kmod-gpio-button-hotplug kmod-leds-gpio kmod-ledtrig-default-on kmod-ledtrig-netdev kmod-ledtrig-timer kmod-nf-nathelper kmod-scsi-core kmod-usb-ohci kmod-usb-storage kmod-usb-uhci kmod-usb2 libc libgcc mtd netifd nvram odhcp6c odhcpd ppp ppp-mod-pppoe swconfig uci wpad-mini
echo

rm -rf /home/bananos/Projects/openwrt_dir320/image_builder/OpenWrt-ImageBuilder-15.05.1-brcm47xx-legacy.Linux-x86_64/build_dir/target-mipsel_mips32_uClibc-0.9.33.2/root-brcm47xx
.....

Note the - symbol before some of the packages, this means that those must be excluded from resulting build. As a result our target image will have almost nothing except external usb support and bare linux kernel. This is due to really tiny capacity of the internal router storage(4Mb). In order to increase it we’ll be using external USB storage(extroot) which is covered in detail in the next sections of this article.

Make sure you have active connection to internet as Image Builder will download some packages from the online repository. At the end of the build, you should be able to find a bunch of prepackaged firmwares in bin/ directory:

bananos@ubby:$ ls bin/brcm47xx/ | grep "asus"

openwrt-15.05.1-brcm47xx-legacy-asus-wl-300g-squashfs.trx
openwrt-15.05.1-brcm47xx-legacy-asus-wl-320gp-squashfs.trx
openwrt-15.05.1-brcm47xx-legacy-asus-wl-330ge-squashfs.trx
openwrt-15.05.1-brcm47xx-legacy-asus-wl-500gp-v1-squashfs.trx
openwrt-15.05.1-brcm47xx-legacy-asus-wl-500gp-v2-squashfs.trx
openwrt-15.05.1-brcm47xx-legacy-asus-wl-500w-squashfs.trx
openwrt-15.05.1-brcm47xx-legacy-asus-wl-520gu-squashfs.trx
openwrt-15.05.1-brcm47xx-legacy-asus-wl-550ge-squashfs.trx
openwrt-15.05.1-brcm47xx-legacy-asus-wl-hdd25-squashfs.trx

Our file of interest is openwrt-15.05.1-brcm47xx-legacy-asus-wl-500gp-v2-squashfs.trx since Asus WL500gp-V2 uses the same hardware as Dlink DIR-320 Rev A.

Firmware Flashing

There are a lot of firmware flashing instructions on the net, but we’ll be using the most generic and reliable via TFTP. Create the following bash file named flash.sh:

#!/bin/bash
echo "=================================================================="
echo "This script will upload dd-wrt firmware (firmware.bin)"
echo "in the current directory to 192.168.0.1 "
echo "during the router's bootup. "
echo ""
echo "* Set your ethernet card's settings to: "
echo "     IP:      192.168.0.10 "
echo "     Mask:    255.255.255.0 "
echo "     Gateway: 192.168.0.1 "
echo "* Unplug the router's power cable. "
echo ""
echo "Press Ctrl+C to abort or any other key to continue... "

read

echo ""
echo "* Re-plug the router's power cable. "
echo ""
echo "=================================================================="
echo "Waiting for the router... Press Ctrl+C to abort. "
echo ""

try(){
ping -c 1 -w 1 192.168.0.1
}
try

while [ "$?" != "0" ] ;
do
try
done
echo "*** Start Flashing **** "
atftp --no-source-port-checking -p -l firmware.bin 192.168.0.1
echo "Firmware successfully loaded!"

The script’s usage instructions are pretty self-explanatory, but before using it we must do a few more things:

bananos@ubby:$ chmod +x flash.sh
bananos@ubby:$ cp bin/brcm47xx/openwrt-15.05.1-brcm47xx-legacy-asus-wl-500gp-v2-squashfs.trx firmware.bin
bananos@ubby:$ sudo apt-get install -y atftp atftpd

Assuming that your USB-to-Serial adapter was detected as /dev/ttyUSB0, let’s open serial terminal to be able to observe what is happening on the router in real time:

bananos@ubby:$ sudo bash -c "screen /dev/ttyUSB0 115200"

Now, prepare your ethernet network card settings and

plug port #1 of your router to linux machine
run ./flash.sh and press any key to see a lot of connect: Network is unreachable messages
plug router power cable and wait for the flashing process to finish

In the serial console you’ll notice something similar to this:

Decompressing...........done

CFE version 1.0.37 for BCM947XX (32bit,SP,LE)
Build Date: Thu Jul 19 10:53:24 CST 2007 (root@redhat9)
Copyright (C) 2000,2001,2002,2003 Broadcom Corporation.

Initializing Arena
Initializing Devices.
Boot partition size = 262144(0x40000)
et0: Broadcom BCM47xx 10/100 Mbps Ethernet Controller 4.150.3.0
CPU type 0x29029: 240MHz
Total memory: 32768 KBytes

Total memory used by CFE:  0x80300000 - 0x803C0720 (788256)
Initialized Data:          0x80337730 - 0x8033AE50 (14112)
BSS Area:                  0x8033AE50 - 0x8033C720 (6352)
Local Heap:                0x8033C720 - 0x803A0720 (409600)
Stack Area:                0x803A0720 - 0x803C0720 (131072)
Text (code) segment:       0x80300000 - 0x80337730 (227120)
Boot area (physical):      0x003C1000 - 0x00401000
Relocation Factor:         I:00000000 - D:00000000

alpha bootcode ver 0.1.2
cmd = ifconfig eth0 -addr=192.168.0.1 -mask=255.255.255.0
We disable the arp for self.
We disable the arp for self.
Device eth0:  hwaddr 00-90-4C-C0-00-01, ipaddr 192.168.0.1, mask 255.255.255.0
        gateway not set, nameserver not set
cmd = go;
cmd = load -raw -addr=0x803c0720 -max=0x3a0000 :
Loader:raw Filesys:tftp Dev:eth0 File:: Options:(null)
Loading: entry point=803C0720
........... 3674112 bytes read
Entry at 0x803c0720
cmd = flash -noheader -mem -size=0x381000 0x803c0720 flash1.trx
Reading from 0x803c0720: Programming...done. 3674112 bytes written
Dest=80001000, source =803c0720, filename=flash1.trx
cmd = boot -raw -z -addr=0x80001000 -max=0x3a0000 flash0.os:
Loader:raw Filesys:raw Dev:flash0.os File: Options:(null)
Loading: entry point=80001000
.. 3712 bytes read
Entry at 0x80001000
Closing network.
Starting program at 0x80001000
[    0.000000] Linux version 3.18.23 (buildbot@builder1) (gcc version 4.8.3 (OpenWrt/Linaro GCC 4.8-2014.04 r48532) ) #1 Tue Mar 1 09:18:44 CET 2016
[    0.000000] CPU0 revision is: 00029029 (Broadcom BMIPS3300)
[    0.000000] bcm47xx: Using ssb bus
[    0.000000] ssb: Found chip with id 0x5354, rev 0x03 and package 0x00
[    0.000000] can not parse nvram name (null)boardnum(null) with value  got -22
[    0.000000] can not parse nvram name (null)rxpo2g(null) with value 0xfff8 got -34
[    0.000000] ssb: Sonics Silicon Backplane found at address 0x18000000
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 02000000 @ 00000000 (usable)
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x00000000-0x01ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00000000-0x01ffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x01ffffff]
[    0.000000] Primary instruction cache 16kB, VIPT, 4-way, linesize 16 bytes.
[    0.000000] Primary data cache 16kB, 2-way, VIPT, cache aliases, linesize 16 bytes
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 8128
[    0.000000] Kernel command line:  noinitrd console=ttyS0,115200
[    0.000000] PID hash table entries: 128 (order: -3, 512 bytes)
[    0.000000] Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Memory: 28956K/32768K available (2605K kernel code, 103K rwdata, 316K rodata, 164K init, 267K bss, 3812K reserved)
[    0.000000] NR_IRQS:128
[    0.000000] MIPS: machine is Unknown Board
[    0.060000] Calibrating delay loop... 239.61 BogoMIPS (lpj=1198080)
[    0.070000] pid_max: default: 32768 minimum: 301
[    0.070000] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.070000] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.080000] NET: Registered protocol family 16
[    0.100000] Switched to clocksource MIPS
[    0.110000] NET: Registered protocol family 2
[    0.110000] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    0.110000] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    0.110000] TCP: Hash tables configured (established 1024 bind 1024)
[    0.110000] TCP: reno registered
[    0.110000] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.110000] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.110000] NET: Registered protocol family 1
[    0.110000] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.120000] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.120000] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.120000] msgmni has been set to 56
[    0.120000] io scheduler noop registered
[    0.120000] io scheduler deadline registered (default)
[    0.120000] Serial: 8250/16550 driver, 16 ports, IRQ sharing enabled
[    0.130000] console [ttyS0] disabled
[    0.150000] serial8250.0: ttyS0 at MMIO 0xb8000300 (irq = 3, base_baud = 1562500) is a U6_16550A
[    0.450000] console [ttyS0] enabled
[    0.470000] serial8250.0: ttyS1 at MMIO 0xb8000400 (irq = 3, base_baud = 1562500) is a U6_16550A
[    0.490000] physmap platform flash device: 02000001 at 1c000000
[    0.490000] physmap-flash.0: Found 1 x16 devices at 0x0 in 8-bit bank. Manufacturer ID 0x000001 Chip ID 0x0000f6
[    0.500000] Amd/Fujitsu Extended Query Table at 0x0040
[    0.510000]   Amd/Fujitsu Extended Query version 1.1.
[    0.510000] physmap-flash.0: Swapping erase regions for top-boot CFI table.
[    0.520000] number of CFI chips: 1
[    0.530000] 6 bcm47xxpart partitions found on MTD device physmap-flash.0
[    0.540000] Creating 6 MTD partitions on "physmap-flash.0":
[    0.540000] 0x000000000000-0x000000040000 : "boot"
[    0.550000] 0x000000040000-0x0000003f0000 : "firmware"
[    0.560000] 0x00000004001c-0x00000004090c : "loader"
[    0.570000] 0x00000004090c-0x000000146000 : "linux"
[    0.580000] 0x000000146000-0x0000003f0000 : "rootfs"
[    0.580000] mtd: device 4 (rootfs) set to be root filesystem
[    0.590000] 1 squashfs-split partitions found on MTD device rootfs
[    0.600000] 0x000000380000-0x0000003f0000 : "rootfs_data"
[    0.600000] 0x0000003f0000-0x000000400000 : "nvram"
[    0.710000] libphy: Fixed MDIO Bus: probed
[    0.710000] b44: Broadcom 44xx/47xx 10/100 PCI ethernet driver version 2.0
[    0.730000] libphy: b44_eth_mii: probed
[    0.850000] b53_common: found switch: BCM5325, rev 0
[    0.860000] b44 ssb0:0: attached PHY driver [Broadcom B53 (1)] (mii_bus:phy_addr=1:1e)
[    0.870000] b44 ssb0:0 eth0: Broadcom 44xx/47xx 10/100 PCI ethernet driver 00:90:4c:c0:00:01
[    0.880000] bcm47xx-wdt bcm47xx-wdt.0: BCM47xx Watchdog Timer enabled (30 seconds, Software Timer)
[    0.890000] GPIO_WDT: failed to register misc device
[    0.890000] TCP: cubic registered
[    0.890000] NET: Registered protocol family 17
[    0.900000] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[    0.910000] Bridge firewalling registered
[    0.920000] 8021q: 802.1Q VLAN Support v1.8
[    0.940000] VFS: Mounted root (squashfs filesystem) readonly on device 31:4.
[    0.950000] Freeing unused kernel memory: 164K (802f7000 - 80320000)
[    2.620000] init: Console is alive
[    2.630000] init: - watchdog -
[    5.430000] usbcore: registered new interface driver usbfs
[    5.430000] usbcore: registered new interface driver hub
[    5.440000] usbcore: registered new device driver usb
[    5.640000] SCSI subsystem initialized
[    5.670000] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    5.680000] ehci-platform: EHCI generic platform driver
[    5.680000] ehci-platform ehci-platform.0: EHCI Host Controller
[    5.690000] ehci-platform ehci-platform.0: new USB bus registered, assigned bus number 1
[    5.700000] ehci-platform ehci-platform.0: irq 6, io mem 0x18003800
[    5.730000] ehci-platform ehci-platform.0: USB 2.0 started, EHCI 1.00
[    5.730000] hub 1-0:1.0: USB hub found
[    5.740000] hub 1-0:1.0: 2 ports detected
[    5.750000] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    5.760000] ohci-platform: OHCI generic platform driver
[    5.770000] ohci-platform ohci-platform.0: Generic Platform OHCI controller
[    5.780000] ohci-platform ohci-platform.0: new USB bus registered, assigned bus number 2
[    5.790000] ohci-platform ohci-platform.0: irq 6, io mem 0x18003000
[    5.850000] hub 2-0:1.0: USB hub found
[    5.860000] hub 2-0:1.0: 2 ports detected
[    5.870000] uhci_hcd: USB Universal Host Controller Interface driver
[    5.890000] usbcore: registered new interface driver usb-storage
[    6.680000] init: - preinit -
[    7.170000] b44 ssb0:0 eth0: Link is up at 100 Mbps, half duplex
[    7.180000] b44 ssb0:0 eth0: Flow control is off for TX and off for RX
[    7.460000] random: mktemp urandom read with 11 bits of entropy available
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
[    8.980000] b44 ssb0:0 eth0: Link is Up - 100Mbps/Full - flow control off
[   11.010000] mount_root: loading kmods from internal overlay
[   11.380000] block: attempting to load /etc/config/fstab
[   11.390000] block: unable to load configuration (fstab: Entry not found)
[   11.400000] block: no usable configuration
[   11.400000] mount_root: jffs2 not ready yet, using temporary tmpfs overlay
[   11.460000] b44 ssb0:0 eth0: powering down PHY
[   11.490000] procd: - early -
[   11.500000] procd: - watchdog -
[   12.000000] b44 ssb0:0 eth0: Link is Down
[   12.410000] procd: - ubus -
[   13.430000] procd: - init -
Please press Enter to activate this console.
[   17.510000] NET: Registered protocol family 10
[   17.530000] Loading modules backported from Linux version master-2015-03-09-0-g141f155
[   17.540000] Backport generated by backports.git backports-20150129-0-gdd4a670
[   17.560000] nf_conntrack version 0.5.0 (455 buckets, 1820 max)
[   17.670000] xt_time: kernel timezone is -0000
[   17.730000] cfg80211: Calling CRDA to update world regulatory domain
[   17.740000] ip_tables: (C) 2000-2006 Netfilter Core Team
[   17.770000] cfg80211: World regulatory domain updated:
[   17.770000] cfg80211:  DFS Master region: unset
[   17.780000] cfg80211:   (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time)
[   17.790000] cfg80211:   (2402000 KHz - 2472000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
[   17.800000] cfg80211:   (2457000 KHz - 2482000 KHz @ 20000 KHz, 92000 KHz AUTO), (N/A, 2000 mBm), (N/A)
[   17.810000] cfg80211:   (2474000 KHz - 2494000 KHz @ 20000 KHz), (N/A, 2000 mBm), (N/A)
[   17.820000] cfg80211:   (5170000 KHz - 5250000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (N/A)
[   17.830000] cfg80211:   (5250000 KHz - 5330000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (0 s)
[   17.840000] cfg80211:   (5490000 KHz - 5730000 KHz @ 160000 KHz), (N/A, 2000 mBm), (0 s)
[   17.840000] cfg80211:   (5735000 KHz - 5835000 KHz @ 80000 KHz), (N/A, 2000 mBm), (N/A)
[   17.850000] cfg80211:   (57240000 KHz - 63720000 KHz @ 2160000 KHz), (N/A, 0 mBm), (N/A)
[   18.230000] PPP generic driver version 2.4.2
[   18.240000] NET: Registered protocol family 24
[   18.410000] b43-phy0: Broadcom 5354 WLAN found (core revision 13)
[   18.440000] b43-phy0: Found PHY: Analog 6, Type 5 (LP), Revision 0
[   18.450000] b43-phy0: Found Radio: Manuf 0x17F, ID 0x2062, Revision 1, Version 0
[   18.470000] Broadcom 43xx driver loaded [ Features: PNL ]
[   18.550000] Broadcom 43xx-legacy driver loaded [ Features: PLD ]
[   26.610000] b44 ssb0:0 eth0: Link is up at 100 Mbps, full duplex
[   26.610000] b44 ssb0:0 eth0: Flow control is off for TX and off for RX
[   28.030000] b44 ssb0:0 eth0: Link is Up - 100Mbps/Full - flow control off
[   31.430000] jffs2_scan_eraseblock(): End of filesystem marker found at 0x0
[   31.450000] jffs2_build_filesystem(): unlocking the mtd device... done.
[   31.450000] jffs2_build_filesystem(): erasing all blocks after the end marker... ^[[A^[[A
[   34.450000] b44 ssb0:0 eth0: powering down PHY
[   34.530000] b44 ssb0:0 eth0: Link is up at 100 Mbps, full duplex
[   34.540000] b44 ssb0:0 eth0: Flow control is off for TX and off for RX
[   34.580000] device eth0.1 entered promiscuous mode
[   34.580000] device eth0 entered promiscuous mode
[   34.600000] br-lan: port 1(eth0.1) entered forwarding state
[   34.610000] br-lan: port 1(eth0.1) entered forwarding state
[   35.830000] done.
[   35.830000] jffs2: notice: (845) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[   36.610000] br-lan: port 1(eth0.1) entered forwarding state
[   80.480000] random: nonblocking pool is initialized

BusyBox v1.23.2 (2016-03-01 08:53:11 CET) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 CHAOS CALMER (15.05.1, r48532)
 -----------------------------------------------------
  * 1 1/2 oz Gin            Shake with a glassful
  * 1/4 oz Triple Sec       of broken ice and pour
  * 3/4 oz Lime Juice       unstrained into a goblet.
  * 1 1/2 oz Orange Juice
  * 1 tsp. Grenadine Syrup
 -----------------------------------------------------
root@OpenWrt:/#

After successful flash procedure, reboot your router by powering it off and setting your ethernet network card back to DHCP (that’s necessary because the default router DHCP server assigns addresses in 192.168.1.XXX subnet, which is different from 192.168.0.XXX which we used for flashing purposes)

Turn on the device once again and wait until wired connection with your host system establishes. After that, you may happily telnet into the router:

bananos@ubby:$ telnet 192.168.1.1
Trying 192.168.1.1...
Connected to 192.168.1.1.
Escape character is '^]'.
 === IMPORTANT ============================
  Use 'passwd' to set your login password
  this will disable telnet and enable SSH
 ------------------------------------------

BusyBox v1.23.2 (2016-03-01 08:53:11 CET) built-in shell (ash)

.....

Congrats! The first, most important step is completed. In the next section we’ll configure extroot to be able to install additional OpenWrt packages such as LuCI and various kernel modules.

Setting up extroot

Connect your USB drive to linux host system. The first thing we’re going to do is partitioning. Assuming that your newly inserted disk is /dev/sdb I would suggest somewhat similar to this:

bananos@ubby:$ sudo fdisk -l /dev/sdb
Disk /dev/sdb: 28 GiB, 30005821440 bytes, 58605120 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa58948b4

Device     Boot   Start      End  Sectors   Size Id Type
/dev/sdb1            63  2008124  2008062 980.5M 82 Linux swap / Solaris
/dev/sdb2       2008125  6024374  4016250   1.9G 83 Linux
/dev/sdb3       6024375 58605119 52580745  25.1G 83 Linux

In my case, I’m using 1.8 inch TOSHIBA MK3008GAL hard drive. To make partitioning table similar to above mentioned you may use GUI tool called GParted or use plain fdisk. Primary partition usage is as follows:

/dev/sdb1 is for swapping purposes due to a low (32Mb) amount of RAM
/dev/sdb2 is for root filesystem (rootfs /)
/dev/sdb3 is for home directory (/home)

After partitioning, format rootfs and /home into ext4 filesystems:

bananos@ubby:$ sudo umount /dev/sdb2
bananos@ubby:$ sudo mkfs.ext4 /dev/sdb2
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 502031 4k blocks and 125696 inodes
Filesystem UUID: 69b17872-9486-4113-8d9f-59a5e4221027
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

bananos@ubby:$ sudo umount /dev/sdb3
bananos@ubby:$ sudo mkfs.ext4 /dev/sdb3

....

bananos@ubby:$ sudo mkswap /dev/sdb1
Setting up swapspace version 1, size = 980.5 MiB (1028120576 bytes)
no label, UUID=457a57b6-aae0-410a-8832-4e182e15e04b

Now, connect USB drive to your router, turn it on, and telnet once again:

root@OpenWrt:/# logread | grep "sda"
Tue Mar  1 08:15:13 2016 kern.notice kernel: [    7.290000] sd 0:0:0:0: [sda] 58605120 512-byte logical blocks: (30.0 GB/27.9 GiB)
Tue Mar  1 08:15:13 2016 kern.notice kernel: [    7.320000] sd 0:0:0:0: [sda] Write Protect is off
Tue Mar  1 08:15:13 2016 kern.debug kernel: [    7.320000] sd 0:0:0:0: [sda] Mode Sense: 03 00 00 00
Tue Mar  1 08:15:13 2016 kern.err kernel: [    7.330000] sd 0:0:0:0: [sda] No Caching mode page found
Tue Mar  1 08:15:13 2016 kern.err kernel: [    7.330000] sd 0:0:0:0: [sda] Assuming drive cache: write through
Tue Mar  1 08:15:13 2016 kern.info kernel: [    7.400000]  sda: sda1 sda2 sda3
Tue Mar  1 08:15:13 2016 kern.notice kernel: [    7.440000] sd 0:0:0:0: [sda] Attached SCSI disk

We’ve just used logread to make sure that hard drive was successfully detected by kernel. Now, let’s proceed to creating external rootfs:

root@OpenWrt:/# ls -l /dev/sda
brw-r--r--    1 root     root        8,   0 Jan  1  1970 /dev/sda
root@OpenWrt:/# block detect
config 'global'
        option  anon_swap       '0'
        option  anon_mount      '0'
        option  auto_swap       '1'
        option  auto_mount      '1'
        option  delay_root      '5'
        option  check_fs        '0'

config 'swap'
        option  uuid    '457a57b6-aae0-410a-8832-4e182e15e04b'
        option  enabled '0'

config 'mount'
        option  target  '/mnt/sda2'
        option  uuid    '69b17872-9486-4113-8d9f-59a5e4221027'
        option  enabled '0'

config 'mount'
        option  target  '/mnt/sda3'
        option  uuid    '37582519-c560-45d3-9a4e-9a95d255b33a'
        option  enabled '0'

We’ve just seen that usb hard drive is present at /dev/sda and has 3 partitions. Next, we’ll create a so-called pivot-overlay:

root@OpenWrt:/# mount /dev/sda2 /mnt ; tar -C /overlay -cvf - . | tar -C /mnt -xf - ; umount /mnt
./
./.fs_state
./work/
./work/work/
./upper/
./upper/etc/
./upper/etc/uci-defaults/
./upper/etc/uci-defaults/11_migrate-sysctl
./upper/etc/uci-defaults/03_network_migration
./upper/etc/uci-defaults/odhcpd.defaults
./upper/etc/uci-defaults/10_migrate-shadow
./upper/etc/uci-defaults/10-fstab
./upper/etc/uci-defaults/12_network-generate-ula
./upper/etc/uci-defaults/09_fix_crc
./upper/etc/config/
./upper/etc/config/dhcp
./upper/etc/config/fstab
./upper/etc/config/firewall
./upper/etc/config/network
./upper/etc/config/system
./upper/etc/config/wireless
./upper/etc/ethers
./upper/etc/dnsmasq.time

The above mentioned command will mount /dev/sda2 as /mnt, then will copy everything from /overlay to /mnt and unmount partition. Now, create fstab template:

root@OpenWrt:/# block detect > /etc/config/fstab ; vi /etc/config/fstab

After editing your fstab file it should look like following:

config 'global'
        option  anon_swap       '0'
        option  anon_mount      '0'
        option  auto_swap       '1'
        option  auto_mount      '1'
        option  delay_root      '5'
        option  check_fs        '1'

config 'swap'
        option  enabled '1'
        option  device '/dev/sda1'

config 'mount'
        option  target  '/overlay'
        option  enabled '1'
        option  device  '/dev/sda2'
        option  fstype  'ext4'
        option  options 'rw,relatime,data=ordered,async,errors=remount-ro'
        option  enabled_fsck '1'

config 'mount'
        option  target  '/home'
        option  enabled '1'
        option  device '/dev/sda3'
        option  fstype  'ext4'
        option  options 'rw,relatime,data=ordered,async,errors=remount-ro'
        option  enabled_fsck '1'

Don’t forget to create /home directory by entering mkdir -p /home! Now, reboot your router and check that everything was mounted correctly:

root@OpenWrt:/# df -h
Filesystem                Size      Used Available Use% Mounted on
rootfs                    1.9G      2.9M      1.7G   0% /
/dev/root                 2.3M      2.3M         0 100% /rom
tmpfs                    14.2M     44.0K     14.2M   0% /tmp
/dev/sda2                 1.9G      2.9M      1.7G   0% /overlay
overlayfs:/overlay        1.9G      2.9M      1.7G   0% /
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sda3                24.6G     43.9M     23.2G   0% /home

root@OpenWrt:/# swapon -s
Filename                                Type            Size    Used    Priority
/dev/sda1                               partition       1004024 0       -1

We can see here, that rootfs(/) partition now has 1.7G of free space, and there’s almost 998M of swap. That’s it! External root filesystem is ready to use. In the next step, we’ll install opkg package manager, to be able to do something useful with our fresh linux system.

Installation of package manager (opkg)

A bare linux system is of no use if we’re not able to install additional software. That’s what we’ll accomplish in this section. Since we don’t have access to internet(yet) from the router itself, we’ll download opkg binary package manually and install it on the working system.

As an additional option you may consider downloading all of the OpenWRT binary packages to newly prepared usb drive if you have plenty of free space. That’s very useful if there will be no access to internet from the router itself during spontaneous connection failures or maintenance periods.

Re-attach hard drive to linux host system and assuming that
/media/bananos/69b17872-9486-4113-8d9f-59a5e4221027 is your /home directory mentioned above, download all packages:

root@ubby:$ cd `/media/bananos/69b17872-9486-4113-8d9f-59a5e4221027` && mkdir repo && cd repo
root@ubby:$ wget -r --no-parent --reject "index.html*" "https://downloads.openwrt.org/chaos_calmer/15.05.1/brcm47xx/legacy/packages/"
......

FINISHED --2016-03-31 16:48:54--
Total wall clock time: 7m 23s
Downloaded: 4508 files, 307M in 2m 9s (2.39 MB/s)

This will download all packages from https://downloads.openwrt.org/chaos_calmer/15.05.1/brcm47xx/legacy/packages/ preserving directory structure. Let’s make it more clean:

root@ubby:$ mv downloads.openwrt.org/chaos_calmer/15.05.1/brcm47xx/legacy/packages/* .
root@ubby:$ rm -rf downloads.openwrt.org/
root@ubby:# ls
base luci  management  packages  routing  telephony

Now, let’s install opkg manager by copying corresponding package to rootfs:

root@ubby:$ cp base/opkg_9c97d5ecd795709c8584e972bfdf3aee3a5b846d-9_brcm47xx.ipk /media/bananos/37582519-c560-45d3-9a4e-9a95d255b33a/opkg.ipk
root@ubby:$ cd /media/bananos/37582519-c560-45d3-9a4e-9a95d255b33a/

root@ubby:$ 7z x opkg.ipk

7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,2 CPUs)

Processing archive: opkg.ipk
Extracting  opkg
Everything is Ok
Size:       71680
Compressed: 64546

root@ubby:$ tar -xvf opkg
./debian-binary
./data.tar.gz
./control.tar.gz

root@ubby:$ rm control.tar.gz opkg.ipg debian-binary rm opkg

Unplug USB-hdd from host system and replug it back to router, run telnet and check that data.tar.gz is at /overlay:

root@OpenWrt:/# ls /overlay/
data.tar.gz  etc          lost+found   upper        work
root@OpenWrt:/# mv /overlay/data.tar.gz /
root@OpenWrt:/# tar -xzvf data.tar.gz
./
./lib/
./lib/upgrade/
./lib/upgrade/keep.d/
./lib/upgrade/keep.d/opkg
./bin/
./bin/opkg
./usr/
./usr/lib/
./usr/lib/opkg/
./usr/sbin/
./usr/sbin/opkg-key
./etc/
./etc/uci-defaults/
./etc/uci-defaults/20_migrate-feeds
./etc/opkg/
./etc/opkg/customfeeds.conf
./etc/opkg/distfeeds.conf
./etc/opkg.conf

If everything was correct and you extracted tar package directly to / then it is possible to run opkg itself:

root@OpenWrt:/# opkg --version
opkg version 0.1.8

Now, if you have working internet connection it is possible to install additional OpenWRT packages like LuCI, but we’re going to leverage our offline repo we’ve created above. Open /etc/opkg/distfeeds.conf and create following file contents:

#src/gz chaos_calmer_base http://downloads.openwrt.org/chaos_calmer/15.05.1/brcm47xx/legacy/packages/base
#src/gz chaos_calmer_luci http://downloads.openwrt.org/chaos_calmer/15.05.1/brcm47xx/legacy/packages/luci
#src/gz chaos_calmer_packages http://downloads.openwrt.org/chaos_calmer/15.05.1/brcm47xx/legacy/packages/packages
#src/gz chaos_calmer_routing http://downloads.openwrt.org/chaos_calmer/15.05.1/brcm47xx/legacy/packages/routing
#src/gz chaos_calmer_telephony http://downloads.openwrt.org/chaos_calmer/15.05.1/brcm47xx/legacy/packages/telephony
#src/gz chaos_calmer_management http://downloads.openwrt.org/chaos_calmer/15.05.1/brcm47xx/legacy/packages/management

src/gz chaos_calmer_base file:///home/repo/base
src/gz chaos_calmer_luci file:///home/repo/luci
src/gz chaos_calmer_packages file:///home/repo/packages
src/gz chaos_calmer_routing file:///home/repo/routing
src/gz chaos_calmer_telephony file:///home/repo/telephony
src/gz chaos_calmer_management file:///home/repo/management

run update procedure:

root@OpenWrt:/# opkg update
Downloading file:///home/repo/base/Packages.gz.
Updated list of available packages in /var/opkg-lists/chaos_calmer_base.
Downloading file:///home/repo/base/Packages.sig.
Signature check passed.
Downloading file:///home/repo/luci/Packages.gz.
Updated list of available packages in /var/opkg-lists/chaos_calmer_luci.
Downloading file:///home/repo/luci/Packages.sig.
Signature check passed.
Downloading file:///home/repo/packages/Packages.gz.
Updated list of available packages in /var/opkg-lists/chaos_calmer_packages.
Downloading file:///home/repo/packages/Packages.sig.
Signature check passed.
Downloading file:///home/repo/routing/Packages.gz.
Updated list of available packages in /var/opkg-lists/chaos_calmer_routing.
Downloading file:///home/repo/routing/Packages.sig.
Signature check passed.
Downloading file:///home/repo/telephony/Packages.gz.
Updated list of available packages in /var/opkg-lists/chaos_calmer_telephony.
Downloading file:///home/repo/telephony/Packages.sig.
Signature check passed.
Downloading file:///home/repo/management/Packages.gz.
Updated list of available packages in /var/opkg-lists/chaos_calmer_management.
Downloading file:///home/repo/management/Packages.sig.
Signature check passed.

Now, it is possible to install any additional software we like. For example, let’s install nano text editor:

root@OpenWrt:/# opkg install nano
Installing nano (2.4.1-1) to root...
Downloading file:///home/repo/packages/nano_2.4.1-1_brcm47xx.ipk.
Installing libncurses (5.9-2) to root...
Downloading file:///home/repo/base/libncurses_5.9-2_brcm47xx.ipk.
Installing terminfo (5.9-2) to root...
Downloading file:///home/repo/base/terminfo_5.9-2_brcm47xx.ipk.
Configuring terminfo.
Configuring libncurses.
Configuring nano.

Make sure internet works

If you have access to an internet via LAN cable (i.e. you do have another router running DHCP in your network with internet access), plug your DIR-320 WAN port into your network. Make sure, it gets dynamic IP:

root@OpenWrt:/# ifconfig

br-lan    Link encap:Ethernet  HWaddr 00:90:4C:C0:00:01
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fd47:5a3c:26b2::1/60 Scope:Global
          inet6 addr: fe80::290:4cff:fec0:1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:37 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:3926 (3.8 KiB)

eth0.2    Link encap:Ethernet  HWaddr 00:90:4C:C0:00:01
          inet addr:192.168.0.194  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::290:4cff:fec0:1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:223 errors:0 dropped:0 overruns:0 frame:0
          TX packets:104 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:50169 (48.9 KiB)  TX bytes:9778 (9.5 KiB)

Make sure that your existing network uses different subnet. By default, OpenWRT is configured to use 192.168.1.X subnet, so you should use something like 192.168.0.X or 192.168.2.X. Verify that you’re able to download content from the internet:

root@OpenWrt:/# wget http://httpbin.org/ip -O /dev/null
Connecting to httpbin.org (54.175.219.8:80)
null                 100% |*******************************|    32   0:00:00 ETA

If you’re not able to get any content outside your network, it is most likely a problem with firewall. One of the symptoms of this problem is that when you issue ping command, there will be following output ping: sendmsg: Operation not permitted.

To allow all traffic in any direction(don’t do this in production environment!), do the following:

root@OpenWrt:/# echo "iptables -P INPUT ACCEPT" >> /etc/firewall.user
root@OpenWrt:/# echo "iptables -P OUTPUT ACCEPT" >> /etc/firewall.user
root@OpenWrt:/# echo "iptables -P FORWARD ACCEPT" >> /etc/firewall.user
root@OpenWrt:/# /etc/init.d/firewall restart

LuCI installation

LuCI is a web-based GUI for managing OpenWRT. A lot of essential configuration which is pretty cumbersome to do via command-line could be easily made through LuCI.

Before installing LuCI packages I would strongly suggest setting up kmod-zram and zram-swap packages if you’re not using external HDD swap, but instead rely on slow, error-prone flash memory. Zram is a special linux kernel module which makes part of your RAM compressed and behave like swap.

Now, let’s proceed:

root@OpenWrt:/# opkg update && opkg install --force-overwrite wget ca-certificates coreutils dropbear
root@OpenWrt:/# opkg install ip fdisk usbutils screen htop
root@OpenWrt:/# opkg install luci luci-proto-3g luci-proto-ipv6 luci-proto-openconnect luci-proto-ppp luci-proto-relay

After successfull installation of the above mentioned packages, reboot and plug your linux host into one of the 4 LAN ports of the router. Since LuCI is not yet configured, it is not possible to access it via WAN, therefore we’re going to setup it from the internal router network. Open up your favorite browser and go to http://192.168.1.1/cgi-bin/luci

From now on, you may fully configure your router by setting up root password, enabling Wi-Fi, etc.

3g/CDMA modem configuration

Now, the most interesting part — how do we connect external CDMA modem to the router and make sure it will function as a WAN interface for LAN clients? At the time of this writing, I had in my hands a used Sierra AirCard 250U:

In this part of the article, we’re going to setup it to act as an external WAN interface. First off, disconnect ethernet cable from router’s WAN port and connect modem. Make sure you’re using USB-hub with external power supply, since 3G modems may drain a lot of current from single USB port and you might experience a lot of weird issues.

Open up terminal console using ssh (telnet should be already disabled if you’ve configured router’s root password):

bananos@ubby:$ ssh  root@192.168.1.1
Warning: Permanently added '192.168.1.1' (RSA) to the list of known hosts.
root@192.168.1.1's password:

When CDMA modem is powered on, it is possible to see it in the list of connected usb devices:

root@OpenWrt:~# lsusb
Bus 001 Device 007: ID 1199:0301 Sierra Wireless, Inc.
Bus 001 Device 008: ID 198f:0220 Beceem Communications Inc. BCSM250 WiMAX Adapter
Bus 001 Device 004: ID 0424:2512 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 003: ID 05ac:0221 Apple, Inc. Aluminum Keyboard (ISO)
Bus 001 Device 005: ID 067b:2506 Prolific Technology, Inc. Kaser 8gB micro hard drive
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 002: ID 05ac:1006 Apple, Inc. Hub in Aluminum Keyboard
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

In this example I’m using Apple Keyboard as USB hub :) Aricard 250U may actually work in CDMA/WiMax mode, and is detected as two separate devices Sierra Wireless, Inc. and Beceem Communications Inc. BCSM250 WiMAX Adapter. Now, let’s search for Sierra support in software packages:

root@OpenWrt:~# opkg update
root@OpenWrt:~# opkg list | grep sierra
kmod-usb-net-sierrawireless - 3.18.23-1 - Kernel support for Sierra Wireless devices
kmod-usb-serial-sierrawireless - 3.18.23-1 - Kernel support for Sierra Wireless devices

Luckily, there is kernel support for Sierra devices and we’re going to leverage that! This is exactly the case where modern linux kernel shines in all its beauty. On top of this kernel modules we need to install some other generic stuff:

root@OpenWrt:~# opkg install kmod-usb-serial kmod-usb-serial-option kmod-usb-serial-wwan kmod-usb-serial-ipw
root@OpenWrt:~# opkg install comgt usb-modeswitch coreutils-stty
root@OpenWrt:~# opkg install kmod-usb-net-sierrawireless kmod-usb-serial-sierrawireless
root@OpenWrt:~# reboot

Make sure that sierra kernel support is functioning:

root@OpenWrt:~# lsmod | grep sierra
sierra                  6048  0
sierra_net              5600  0
usbnet                 15872  1 sierra_net

As this point, we have a serial interface to CDMA modem at /dev/ttyUSB0. Verify that modem responds to AT commands:

root@OpenWrt:~# gcom info -d /dev/ttyUSB0
##### Wireless WAN Modem Configuration #####
Product text:
====

Manufacturer: Sierra Wireless, Inc.
Model: AC250U Rev 2.0 (3)
Revision: p3012901,8815 [May 28 2010 14:42:39]
QCOM: SWI6085_FP.01.28
BOOT: SWI6085_PP.01.29.01 2010/05/28 18:13:46
APPL: SWI6085_PP.01.29.01 2010/05/28 18:13:46
USB VID: 0x1199 PID: 0x0301
ESN: 09612160827
0x60B98F3B
+GCAP: +CIS707-A, CIS-856, CIS-856-A, +MS, +ES, +DS, +FCLASS
SKU: 0x132BD1
OK
====
Manufacturer:           ERROR
IMEI and Serial Number: 09612160827
Manufacturer's Revision:
p3012901,8815 [May 28 2010 14:42:3
Hardware Revision:

Network Locked:

Customisation:

Band settings:          (
)
APN:

##### END #####

Now, switch to LuCI and go to Network->Interfaces. Stop WAN6 unless you're going to use IPv6 and edit WAN interface.

When editing existing WAN interface, change Protocol to UMTS/GPRS/EV-DO, click Switch protocol button and wait for additional parameters to appear:

On this screen you should enter provider specific info (take a look at your contract/prepaid plan details). Since my modem is connected to Ukraine's CDMA Intertelecom my connection parameters are:

Protocol: UMTS/GPRS/EV-DO
Modem device: /dev/ttyUSB0 (we've tested above that it actually responds to AT commands)
Service type: CDMA/EV-DO
APN: empty
PIN: empty
PAP/CHAP username: IT
PAP/CHAP password: IT
Dial number: #777

Note, that your interface will now be named 3g-wan and since there is a flag called Bring up on boot in the Advanced tab your CDMA connection will be established automatically on power up. To make sure that everything works, open up terminal once again:

root@OpenWrt:~# logread | grep -E "chat|ppp"
Tue Apr  5 19:38:31 2016 daemon.notice pppd[6810]: pppd 2.4.7 started by root, uid 0
Tue Apr  5 19:38:32 2016 local2.info chat[6821]: abort on (BUSY)
Tue Apr  5 19:38:32 2016 local2.info chat[6821]: abort on (NO CARRIER)
Tue Apr  5 19:38:32 2016 local2.info chat[6821]: abort on (ERROR)
Tue Apr  5 19:38:32 2016 local2.info chat[6821]: abort on (NO DIAL TONE)
Tue Apr  5 19:38:32 2016 local2.info chat[6821]: abort on (NO ANSWER)
Tue Apr  5 19:38:32 2016 local2.info chat[6821]: abort on (DELAYED)
Tue Apr  5 19:38:32 2016 local2.info chat[6821]: report (CONNECT)
Tue Apr  5 19:38:32 2016 local2.info chat[6821]: timeout set to 10 seconds
Tue Apr  5 19:38:32 2016 local2.info chat[6821]: send (AT^M)
Tue Apr  5 19:38:32 2016 local2.info chat[6821]: expect (OK)
Tue Apr  5 19:38:32 2016 local2.info chat[6821]: AT^M^M
Tue Apr  5 19:38:32 2016 local2.info chat[6821]: OK
Tue Apr  5 19:38:32 2016 local2.info chat[6821]:  -- got it
Tue Apr  5 19:38:32 2016 local2.info chat[6821]: send (ATZ^M)
Tue Apr  5 19:38:33 2016 local2.info chat[6821]: timeout set to 30 seconds
Tue Apr  5 19:38:33 2016 local2.info chat[6821]: expect (OK)
Tue Apr  5 19:38:33 2016 local2.info chat[6821]: ^M
Tue Apr  5 19:38:33 2016 local2.info chat[6821]: ATZ^M^M
Tue Apr  5 19:38:33 2016 local2.info chat[6821]: OK
Tue Apr  5 19:38:33 2016 local2.info chat[6821]:  -- got it
Tue Apr  5 19:38:33 2016 local2.info chat[6821]: send (ATDT#777^M)
Tue Apr  5 19:38:33 2016 local2.info chat[6821]: expect (CONNECT)
Tue Apr  5 19:38:33 2016 local2.info chat[6821]: ^M
Tue Apr  5 19:38:33 2016 local2.info chat[6821]: ATDT#777^M^M
Tue Apr  5 19:38:33 2016 local2.info chat[6821]: CONNECT
Tue Apr  5 19:38:33 2016 local2.info chat[6821]:  -- got it
Tue Apr  5 19:38:33 2016 local2.info chat[6821]: send (^M)
Tue Apr  5 19:38:33 2016 daemon.info pppd[6810]: Serial connection established.
Tue Apr  5 19:38:33 2016 daemon.info pppd[6810]: Using interface 3g-wan
Tue Apr  5 19:38:33 2016 kern.info kernel: [ 8554.840000] 3g-wan: renamed from ppp0
Tue Apr  5 19:38:33 2016 daemon.notice pppd[6810]: Connect: 3g-wan < --> /dev/ttyUSB0
Tue Apr  5 19:38:33 2016 daemon.notice pppd[6810]: PAP authentication succeeded
Tue Apr  5 19:38:33 2016 daemon.notice pppd[6810]: local  IP address 10.227.181.163
Tue Apr  5 19:38:33 2016 daemon.notice pppd[6810]: remote IP address 172.17.4.100
Tue Apr  5 19:38:33 2016 daemon.notice pppd[6810]: primary   DNS address 192.168.4.3
Tue Apr  5 19:38:33 2016 daemon.notice pppd[6810]: secondary DNS address 192.168.4.7
Tue Apr  5 20:08:32 2016 daemon.info pppd[6810]: LCP terminated by peer
Tue Apr  5 20:08:32 2016 daemon.info pppd[6810]: Connect time 30.0 minutes.
Tue Apr  5 20:08:32 2016 daemon.info pppd[6810]: Sent 826328 bytes, received 547992 bytes.
Tue Apr  5 20:08:33 2016 daemon.info pppd[6810]: Terminating on signal 15
Tue Apr  5 20:08:35 2016 daemon.notice pppd[6810]: Connection terminated.
Tue Apr  5 20:08:36 2016 daemon.notice pppd[6810]: Modem hangup
Tue Apr  5 20:08:36 2016 daemon.info pppd[6810]: Exit.
Tue Apr  5 20:08:37 2016 daemon.notice pppd[7884]: pppd 2.4.7 started by root, uid 0
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: abort on (BUSY)
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: abort on (NO CARRIER)
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: abort on (ERROR)
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: abort on (NO DIAL TONE)
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: abort on (NO ANSWER)
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: abort on (DELAYED)
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: report (CONNECT)
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: timeout set to 10 seconds
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: send (AT^M)
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: expect (OK)
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: AT^M^M
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: OK
Tue Apr  5 20:08:38 2016 local2.info chat[7896]:  -- got it
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: send (ATZ^M)
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: timeout set to 30 seconds
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: expect (OK)
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: ^M
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: ATZ^M^M
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: OK
Tue Apr  5 20:08:38 2016 local2.info chat[7896]:  -- got it
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: send (ATDT#777^M)
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: expect (CONNECT)
Tue Apr  5 20:08:38 2016 local2.info chat[7896]: ^M
Tue Apr  5 20:08:39 2016 local2.info chat[7896]: ATDT#777^M^M
Tue Apr  5 20:08:39 2016 local2.info chat[7896]: CONNECT
Tue Apr  5 20:08:39 2016 local2.info chat[7896]:  -- got it
Tue Apr  5 20:08:39 2016 local2.info chat[7896]: send (^M)
Tue Apr  5 20:08:39 2016 daemon.info pppd[7884]: Serial connection established.

These are the logs from pppd daemon and CHAT script, responsible for sending AT commands to modem. As you can see there may be random hangups, but the connection is automatically re-established which is very nice.

And when you enter ifconfig command you should see 3g-wan interface too:

root@OpenWrt:~# ifconfig
3g-wan    Link encap:Point-to-Point Protocol
          inet addr:10.227.196.50  P-t-P:172.17.4.100  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:977 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2117 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:529666 (517.2 KiB)  TX bytes:324250 (316.6 KiB)

In case you’re interested in diving deeper into how things work, I’d recommend looking at these files:

/etc/config/network contains network configuration
/lib/netifd/proto/3g.sh shell logic responsible for running pppd daemon

Measuring signal strength

The last problem you’ll probably stumble upon — is how to measure signal strength, especially if you’re setting up standalone solution at a remote location with a modem connected to an external antenna. This is usually accomplished by sending special AT command to modem. Note, however, that most software and examples over internet are targeted towards more popular GRPS/UMTS modems, and they will probably won’t work.

Before using serial protocol to communicate with modem, we must shutdown existing 3g-wan connection, because otherwise we’ll get cat: read error: Resource temporarily unavailable:

root@OpenWrt:~# ifdown wan

Note, that ifdown won’t accept 3g-wan as interface, but oddly enough the above command works. For now, let’s focus on how to send AT commands to modem:

run screen
open 2 consoles

In the first console run cat /dev/ttyUSB0 (this is a place to look for modem replies) and in the second run echo -ne "at!status?\r\n" > /dev/ttyUSB0. Switch back consoles and you should see something like this:

root@OpenWrt:~# cat /dev/ttyUSB0
at!rssi?
-83

OK

A range from -60 dbm to -90 dBm is considered adequate. Using this special AT command you may now run it in a loop with small delay, observe signal strength in the second console and properly adjust external antenna. Also, there seems to be an unofficial OpenWRT package specifically for signal monitoring called 3ginfo, but I’ve not tried it and not sure if it works for CDMA/EVDO modems.

TL;DR;

Seriously, If you want to avoid all the hassle of setting up CDMA modem and OpenWRT, just buy yourself a device which combines WiFi router and 3g modem — MiFi will do just fine :)

Download openwrt-15.05.1-brcm47xx-legacy-asus-wl-500gp-v2-squashfs.trx (MD5: 420e24e9a042c01585d818c14afa11d3)

Running php 5.2 fpm on ARM processor

bananos — Sun, 20 Mar 2016 20:06:57 +0000

Sometimes you need to run legacy php code on ARM processors. Since php 5.2 was released 2 November 2006 at the time there were not a lot of powerful ARM solutions capable of running full-fledged websites. However, right now there is a possibility to run pretty heavy code on modern ARM platforms and a lot of linux distros support ARM architecture. There is even dedicated ARM cloud hosting.

In this post we’re going to fix the inability to run PHP in the FPM(Fast Process Manager) mode which is an absolute requirement if you want to run production websites.

Compiling from sources

Since there is no old php releases in modern linux distros we’re obviously going to compile everything from scratch. For our purposes we’ll use ubuntu 15.04, let’s download php sources from PHP museum:

$ mkdir -p /mnt/build/ && cd /mnt/build/
$ wget http://museum.php.net/php5/php-5.2.17.tar.gz && tar -xf php-5.2.17.tar.gz
$ apt-get install -y build-essential nano wget curl libbz2-dev libncurses5-dev autoconf libxml2-dev

PHP-FPM patch

For historical reasons, Fast Process Manager(FPM) for 5.2.X branch exists only in the form of patch. Let’s download this patch and apply it the most recent 5.2 version:

$ cd php-5.2.17/
$ wget http://php-fpm.org/downloads/php-5.2.17-fpm-0.5.14.diff.gz && gzip -d php-5.2.17-fpm-0.5.14.diff.gz
$ patch -p1 < php-5.2.17-fpm-0.5.14.diff
 patching file configure
Hunk #7 succeeded at 110645 (offset 1324 lines).
Hunk #8 succeeded at 118930 (offset 1324 lines).
Hunk #9 succeeded at 119634 (offset 1324 lines).
Hunk #10 succeeded at 119689 (offset 1324 lines).
patching file configure.in
patching file libevent/ChangeLog
patching file libevent/Makefile.am
patching file libevent/Makefile.in
patching file libevent/README
patching file libevent/aclocal.m4

After this operation --enable-fpm option becomes available in ./configure script:

$ ./configure --help | grep "fpm"
  --enable-fpm              FastCGI: If this is enabled, the fastcgi support
  --with-fpm-conf=PATH        Set the path for php-fpm configuration file [PREFIX/etc/php-fpm.conf]
  --with-fpm-log=PATH         Set the path for php-fpm log file [PREFIX/logs/php-fpm.log]
  --with-fpm-pid=PATH         Set the path for php-fpm pid file [PREFIX/logs/php-fpm.pid]

Let’s try to compile php with FPM support:

$ ./configure --prefix=/opt/php52 --enable-fastcgi --enable-fpm

loading cache ./config.cache
checking for Cygwin environment... no
checking for mingw32 environment... no
checking for egrep... grep -E
checking for a sed that does not truncate output... /bin/sed
checking host system type... armv7l-unknown-linux-gnueabi
checking target system type... armv7l-unknown-linux-gnueabi
checking for gcc... gcc
checking whether the C compiler (gcc  ) works... yes
checking whether the C compiler (gcc  ) is a cross-compiler... no
checking whether we are using GNU C... yes
....

Thank you for using PHP.

So far, everything is looking fine. Let’s try to build it by running make using 4 ARM cores to speedup process:

$ make -j4

/bin/sh /mnt/build/php-5.2.17/libtool --silent --preserve-dup-deps --mode=compile gcc -Iext/date/lib -Iext/date/ -I/mnt/build/php-5.2.17/ext/date/ -DPHP_ATOM_
INC -I/mnt/build/php-5.2.17/include -I/mnt/build/php-5.2.17/main -I/mnt/build/php-5.2.17 -I/mnt/build/php-5.2.17/ext/date/lib -I/usr/include/libxml2 -I/mnt/bu
ild/php-5.2.17/TSRM -I/mnt/build/php-5.2.17/Zend    -I/usr/include -g -O2  -c /mnt/build/php-5.2.17/ext/date/php_date.c -o ext/date/php_date.lo
/bin/sh /mnt/build/php-5.2.17/libtool --silent --preserve-dup-deps --mode=compile gcc -Iext/date/lib -Iext/date/ -I/mnt/build/php-5.2.17/ext/date/ -DPHP_ATOM_
INC -I/mnt/build/php-5.2.17/include -I/mnt/build/php-5.2.17/main -I/mnt/build/php-5.2.17 -I/mnt/build/php-5.2.17/ext/date/lib -I/usr/include/libxml2 -I/mnt/bu
ild/php-5.2.17/TSRM -I/mnt/build/php-5.2.17/Zend    -I/usr/include -g -O2  -c /mnt/build/php-5.2.17/ext/date/lib/astro.c -o ext/date/lib/astro.lo

..........

/mnt/build/php-5.2.17/ext/dom/node.c: In function ‘dom_canonicalization’:
/mnt/build/php-5.2.17/ext/dom/node.c:1953:21: error: dereferencing pointer to incomplete type ‘xmlBuf {aka struct _xmlBuf}’
    ret = buf->buffer->use;
                     ^
Makefile:553: recipe for target 'ext/dom/node.lo' failed
make: *** [ext/dom/node.lo] Error 1

libxml2 patching

And boom! We’re stuck with the first error: dereferencing pointer to incomplete type xmlBuf {aka struct _xmlBuf}. After googling around, the problem seems to be related to libxml2 incompatibility and is well documented. Let’s fix that error by applying yet another patch:

$ wget https://github.com/bananos/dockerized/raw/master/armbuild-php52/patches/libxml29_compat.patch
$ patch -p1 < libxml29_compat.patch
patching file ext/dom/node.c
Hunk #1 succeeded at 1950 (offset 55 lines).
patching file ext/dom/documenttype.c
Hunk #1 succeeded at 215 (offset 10 lines).
patching file ext/simplexml/simplexml.c
Hunk #1 succeeded at 1343 (offset -74 lines).

And run make once again:

$ make -j4

......

In file included from /mnt/build/php-5.2.17/sapi/cgi/fpm/fpm_shm_slots.h:8:0,
                 from /mnt/build/php-5.2.17/sapi/cgi/fpm/fpm_children.c:28:
/mnt/build/php-5.2.17/sapi/cgi/fpm/fpm_atomic.h:62:2: error: #error unsupported processor. please write a patch and send it to me
 #error unsupported processor. please write a patch and send it to me
  ^
/mnt/build/php-5.2.17/sapi/cgi/fpm/fpm_atomic.h:66:32: error: unknown type name ‘atomic_t’
 static inline int fpm_spinlock(atomic_t *lock, int try_once)
                                ^
In file included from /mnt/build/php-5.2.17/sapi/cgi/fpm/fpm_children.c:28:0:
/mnt/build/php-5.2.17/sapi/cgi/fpm/fpm_shm_slots.h:16:3: error: unknown type name ‘atomic_t’
   atomic_t lock;
   ^
In file included from /mnt/build/php-5.2.17/sapi/cgi/fpm/fpm_shm_slots.h:8:0,
                 from /mnt/build/php-5.2.17/sapi/cgi/fpm/fpm_worker_pool.c:15:
/mnt/build/php-5.2.17/sapi/cgi/fpm/fpm_atomic.h:62:2: error: #error unsupported processor. please write a patch and send it to me
 #error unsupported processor. please write a patch and send it to me
  ^
/mnt/build/php-5.2.17/sapi/cgi/fpm/fpm_atomic.h:66:32: error: unknown type name ‘atomic_t’
 static inline int fpm_spinlock(atomic_t *lock, int try_once)
                                ^
In file included from /mnt/build/php-5.2.17/sapi/cgi/fpm/fpm_worker_pool.c:15:0:
/mnt/build/php-5.2.17/sapi/cgi/fpm/fpm_shm_slots.h:16:3: error: unknown type name ‘atomic_t’
   atomic_t lock;
   ^
Makefile:433: recipe for target 'sapi/cgi/fpm/fpm_worker_pool.lo' failed
make: *** [sapi/cgi/fpm/fpm_worker_pool.lo] Error 1
make: *** Waiting for unfinished jobs....
Makefile:431: recipe for target 'sapi/cgi/fpm/fpm_children.lo' failed
make: *** [sapi/cgi/fpm/fpm_children.lo] Error 1

Well, this is embarrassing. As we can see from compiler logs, the FPM author suggests us to write a patch and send it to him because current CPU type is not supported. Obviously, the first thing I did was to try google yet another patch to solve this problem, but unfortunately, there is no one — people moved on to newer PHP versions and no one cared enough to write a fix.

Therefore, we’re going to write our own patch. Let’s take a look at /mnt/build/php-5.2.17/sapi/cgi/fpm/fpm_atomic.h:62:2



  /* $Id: fpm_atomic.h,v 1.3 2008/09/18 23:34:11 anight Exp $ */
  /* (c) 2007,2008 Andrei Nigmatulin */

#ifndef FPM_ATOMIC_H
#define FPM_ATOMIC_H 1

#include 
#include 

#if ( __i386__ || __i386 )

typedef int32_t                     atomic_int_t;
typedef uint32_t                    atomic_uint_t;
typedef volatile atomic_uint_t      atomic_t;

static inline atomic_int_t atomic_fetch_add(atomic_t *value, atomic_int_t add)
{
        __asm__ volatile ( "lock;" "xaddl %0, %1;" :
                "+r" (add) : "m" (*value) : "memory");

        return add;
}

static inline atomic_uint_t atomic_cmp_set(atomic_t *lock, atomic_uint_t old, atomic_uint_t set)
{
        unsigned char res;

        __asm__ volatile ( "lock;" "cmpxchgl %3, %1;" "sete %0;" :
                "=a" (res) : "m" (*lock), "a" (old), "r" (set) : "memory");

    return res;
}

#elif ( __amd64__ || __amd64 )

// definitions of the same functions for AMD64

#else

#error unsupported processor. please write a patch and send it to me

#endif

Atomic types

Looks like we need to define those 2 functions for ARM platform. But how do we do it without proper knowledge of C and specifics of armhf architecture? Luckily, we may steal some code from newer PHP versions, which do work in FPM mode on ARM platform. Go to https://github.com/php/php-src/blob/PHP-5.3/sapi/fpm/fpm/fpm_atomic.h and take a look at source code:



    /* $Id: fpm_atomic.h,v 1.3 2008/09/18 23:34:11 anight Exp $ */
    /* (c) 2007,2008 Andrei Nigmatulin */

#ifndef FPM_ATOMIC_H
#define FPM_ATOMIC_H 1

#if HAVE_INTTYPES_H
# include 
#else
# include 
#endif
#include 

#ifdef HAVE_BUILTIN_ATOMIC

/**
 * all the cases below (as provided by upstream) define:
 * word as atomic_int_t, and
 * unsigned word as atomic_uint_t
 * and only use volatile atomic_uint_t as atomic_t
 */

typedef volatile unsigned long atomic_t;
#define atomic_cmp_set(a,b,c) __sync_bool_compare_and_swap(a,b,c)

#elif ( __i386__ || __i386 )

///  a lot of platform-speicific definitions of above mentioned functions

#if (__GNUC__) && (__GNUC__ < 4 || (__GNUC__ == 4 && __GNUC_MINOR__ < 2))

#elif ( __arm__ || __arm ) /* W-Mark Kubacki */

///  other code we're not interested in

Although, it does contain the mention of ARM, the code defined for this case won’t be executed, since this is not our case:

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/5/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.2.1-22ubuntu2' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-armhf/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-armhf --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-armhf --with-arch-directory=arm --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --enable-multilib --disable-sjlj-exceptions --with-arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb --disable-werror --enable-multilib --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
Thread model: posix
gcc version 5.2.1 20151010 (Ubuntu 5.2.1-22ubuntu2)

What we’re going to do is leveraging built-in atomic types, which are controlled by HAVE_BUILTIN_ATOMIC flag. But first, let’s do a drop-in replacement of fpm_atomic.h from 5.3 branch:

$ cd /mnt/build/php-5.2.17/sapi/cgi/fpm
$ rm fpm_atomic.h && wget "https://github.com/php/php-src/raw/PHP-5.3/sapi/fpm/fpm/fpm_atomic.h"

Another magic step is in fact, fix configure system to be able to enable above mentioned flag. Open /mnt/build/php-5.2.17/sapi/cgi/fpm/acinclude.m4 and add at the top:

AC_DEFUN([AC_FPM_BUILTIN_ATOMIC],
[
  AC_MSG_CHECKING([if gcc supports __sync_bool_compare_and_swap])
  AC_TRY_LINK(,
  [
    int variable = 1;
    return (__sync_bool_compare_and_swap(&variable, 1, 2)
           && __sync_add_and_fetch(&variable, 1)) ? 1 : 0;
  ],
  [
    AC_MSG_RESULT([yes])
    AC_DEFINE(HAVE_BUILTIN_ATOMIC, 1, [Define to 1 if gcc supports __sync_bool_compare_and_swap() a.o.])
  ],
  [
    AC_MSG_RESULT([no])
  ])
])

After this is done, everything should compile just fine.

TL;DR

For those lazy folks who don’t want all the investigation details here’s an updated patch which I prepared:
https://github.com/bananos/dockerized/blob/master/armbuild-php52/patches/php-5.2.17-fpm-arm-0.5.14.diff

In order to use it, just download vanilla php-5.2.17 sources and apply the patch:

mkdir -p /tmp/build
cd /tmp/build && wget http://museum.php.net/php5/php-5.2.17.tar.gz && tar -xf php-5.2.17.tar.gz
cd /tmp/build && wget https://github.com/bananos/dockerized/raw/master/armbuild-php52/patches/php-5.2.17-fpm-arm-0.5.14.diff
cd /tmp/build && patch -p1 -d php-5.2.17/ < /tmp/build/php-5.2.17-fpm-arm-0.5.14.diff

If you’re familiar with Docker you may might also check precompiled and flattened PHP-5.2.17 ARMHF image
with some most popular PECL packages pre-built.

Happy running your old legacy PHP code!

Sources

http://php-fpm.org/

https://github.com/bananos/dockerized/tree/master/armbuild-php52/

https://github.com/php/php-src/

https://code.google.com/archive/p/php52-backports/issues/16

http://saravananlinux.blogspot.com/2014/01/libxml-error-while-compiling-dual-php.html

CloudWatch custom metrics

bananos — Wed, 16 Dec 2015 15:40:08 +0000

In this post we’re going to investigate how to create custom metrics for the AWS CloudWatch service. In some cases it’s not enough to have default set of AWS metrics which are already defined for you in dashboard. Let’s say you want to monitor the number of active customers on your website, or you would like to know the number of failed requests to your API backend.

Whatever it is, CloudWatch will allow you to record any time-based data using AWS APIs.

Install AWS CLI

First off, in order to post data to CloudWatch you need to install and setup command-line tools to work with AWS APIs:

$ pip install awscli

If you don’t know what pip is, head over pip installation manual.
Once AWS CLI is installed the aws command should be available via your command shell:

$ aws
usage: aws [options]   [ ...] [parameters]
To see help text, you can run:

aws help
aws: error: too few arguments

Before issuing real commands to AWS service you need to configure cli tools:

$ aws configure
AWS Access Key ID [****************KCEQ]:
AWS Secret Access Key [****************x++s]:
Default region name [us-west-2]:
Default output format [json]:

After successfull configuration you should be able to use command-line interface for AWS, for example, let’s list all available EC2 instances:

$ aws ec2 describe-instances | grep "PublicDnsName" | awk '{$1=$1};1' | uniq
"PublicDnsName": "ec2-44-148-96-212.us-west-2.compute.amazonaws.com",
"PublicDnsName": "ec2-42-25-174-143.us-west-2.compute.amazonaws.com",
"PublicDnsName": "ec2-44-68-194-93.us-west-2.compute.amazonaws.com",
"PublicDnsName": "ec2-44-69-229-202.us-west-2.compute.amazonaws.com",

Choose a metric

Let’s say that we want to monitor application API status by periodically requesting specific endpoint and checking HTTP response code. In our example we’ll use http://httpbin.org/status/200 URL to demo.

We will consider HTTP 200 response code as a service alive event and treat it as 1, and any other response code(including timeout) as a service failure event or 0.

The following bash code(service_mon.sh) will output HTTP response code:

#!/bin/bash
CODE=`curl -k -s -o /dev/null -w "%{http_code}" http://httpbin.org/status/200`
echo $CODE

Run it and verify that script outputs 200:

$ ./service_mon.sh
200

Choose metric name

We’re going to name our metric ServiceStatus and will report it to CloudWatch using cloudwatch subcommand. Modify above mentioned service_mon.sh script to be the following:

#!/bin/bash

# get response code
CODE=`curl -k -s -o /dev/null -w "%{http_code}" http://httpbin.org/status/200`
if [ "$CODE" -eq 200 ] ; then
aws cloudwatch put-metric-data --metric-name ServiceStatus --namespace CL_AGENTS --value 1 --unit "Count"
else
aws cloudwatch put-metric-data --metric-name ServiceStatus --namespace CL_AGENTS --value 0 --unit "Count"
fi

Note CL_AGENTS in cmd parameters, this is a convenient group name for all your metrics, typically you’d want to put your application name in there. In this example we’re specifying a type for our data called Count, this is not the only possible option. Here’s a list of other units:

[ Seconds, Microseconds, Milliseconds, Bytes, Kilobytes, Megabytes, Gigabytes, Terabytes, Bits, Kilobits,
Megabits, Gigabits, Terabits, Percent, Count, Bytes/Second, Kilobytes/Second, Megabytes/Second, Gigabytes/Second,
Terabytes/Second, Bits/Second, Kilobits/Second, Megabits/Second, Gigabits/Second, Terabits/Second, Count/Second, None ]

Periodic script execution

Put the script we’ve written into crontab:

# m h  dom mon dow   command
*/5 * * * * /mnt/service_mon.sh

This will post service status to CloudWatch every 5 minutes. Here’s how it might look like in web based AWS console:

You may also report multiple values at the same time using JSON file:

[
{
"MetricName": "ServiceStatus",
"Timestamp": "Wednesday, June 12, 2013 8:28:20 PM",
"Value": 1,
"Unit": "Count"
},
{
"MetricName": "ServiceStatus",
"Timestamp": "Wednesday, June 12, 2013 8:30:20 PM",
"Value": 0,
"Unit": "Count"
}
]

To do bulk reporting use this format:

$ aws cloudwatch put-metric-data --namespace "CL_AGENTS" --metric-data file://metric.json

Setup alarms

Up until now our metric is almost useless, except that we may see nice graphs in the CloudWatch. To do it more usable, let’s add alarm if our API is down:

Select ServiceName metric in the CL_AGENTS namespace we’ve just created
Click Create Alarm button on the right side of the graph
In the subsequent popup, select alarm details:

Gotchas

Keep in mind that CloudWatch will store your time-series data only for 2 weeks (14 days), therefore if you want more history and insights for your data you’re probably will be better off using custom database like InfluxDB for example
Make sure to setup aws cli under the same user which your crontab is using, otherwise AWS configuration won’t be accessible for cron and your monitoring script will not work

Intellij IDEA and Go plugin

bananos — Wed, 21 Aug 2013 13:39:59 +0000

Ever wondered where to find IDE for programming Google Go language?
Well, there are not a lot of choices out there and some of them lack pretty significant functionality, especially if you’re a newbie in Go like me.

Some popular choices include:

1. LiteIDE the only IDE targeted specifically towards Go
2. GoSublime a Sublime Text plugin

I’ve tried both of them and found out that both of them lack Go to declaration feature, which is crucual in the process of digging around new code. On top of that LiteIDE requires installation of gocode daemon for autocompletion to work fine.

As you can see usage of these IDEs requires a lot of bells and whistles to get basic functioning.

However there’s a third option which I’m going to talk about in this post: Google Go language plugin for IntelliJ Idea

It covers two of my initial requirements for Go IDE:

Go to declaration feature, i.e. you may click on any declaration and go check how it’s implemented under the hood
Autocompletion, that’s a huge time saver and a great helper in exploring available options

If you’re familiar with any Intellij IDEs you’ll get additional time-saving benefits too. Let’s proceed!

As of time of this writing, there’s no up-to-date binary distribution of the plugin (hence, the post:),
so we’re going to build everything from the sources.

Install IDEA community edition

Go to Jetbrain download page and install latest free community edition.
As of time of this writing the latest IDEA version was 12.

Grab IDEA sources!

Get Intellij IDEA community sources

# repo is ~ 1gb, this may take a while
git clone git://git.jetbrains.org/idea/community.git idea
cd idea

# figure out what IDEA build you've just installed
cat /Applications/IntelliJ\ IDEA\ 12.app/build.txt; echo -e "\r\n";
IC-129.713

# checkout specified tag in the source repo
git checkout idea/129.713

# run ant to build everything
ant

Depending on the machine this may take 10-20min to compile.

Grab Go plugin sources

git clone https://github.com/mtoader/google-go-lang-idea-plugin.git
cd google-go-land-idea-plugin/

# compile plugin
ANT_OPTS=-Xmx1g ant -Didea.community.build=/Users/bananos/Projects/idea/ -f build-package.xml

Buildfile: /Users/bananos/Projects/google-go-lang-idea-plugin/build-package.xml

clean:

init:
[mkdir] Created dir: /Users/bananos/Projects/google-go-lang-idea-plugin/build
[mkdir] Created dir: /Users/bananos/Projects/google-go-lang-idea-plugin/dist
[echo] Using IDEA build from: /Users/bananos/Projects/idea/

compile:
[javac2] Compiling 561 source files to /Users/bananos/Projects/google-go-lang-idea-plugin/build

........................
........................
package:
[jar] Building jar: /Users/bananos/Projects/google-go-lang-idea-plugin/dist/ro.redeul.google.go.jar

BUILD SUCCESSFUL
Total time: 23 seconds

Make sure you specify correct idea.community.build variable which should point to recently checked out IDEA community repo.

Unless you struggled with some critical errors, expect to find *.jar files:

ls -l dist/

total 1904
-rw-r--r--  1 bananos  staff  974789 Aug 21 15:55 ro.redeul.google.go.jar

This is a binary distribution we were looking for!

Install plugin

Run IDEA, and go to Preferences->Plugins and click Install plugin from disk button, then select ro.redeul.google.go.jar file from the previous step.

Install Go SDK

Assuming that you’ve installed Go via brew, your Go SDK will be in /usr/local/Cellar/go/1.1.1/

Go to the File -> Project Structure and configure Go SDK by clicking + button and
selecting /usr/local/Cellar/go/1.1.1/ folder.

Make sure to add your project pkg folder under Classpath and your project sources under Sourcepath tabs.

TL;DR

For lazy ones, here is a link to compiled binary!

Phonegap vs. Titanium

bananos — Mon, 02 Jul 2012 13:59:46 +0000

For a while we’ve been spiking on different solutions to implement mobile app for our platform, and here are some of my initial thoughts on the problem. Given that I’m too lazy and not so enthusiastic about Objective-C for iOS or Java for Android I was searching for some solution which could deliver value from both worlds with minimum initial effort.

Phonegap (hybrid approach)

Pros

Ability to develop apps in HTML5 ( means more web devices are captured)
Overall speed of iterative development is high (it’s almost like refreshing the page)
Easy to debug (remote WebKit console in Emulator)
It’s JavaScript!
Huge amount of JS-frameworks for touch devices (Sencha, iUI, jQuery mobile, iWebkit, etc) ready for use
Ability to make bindings to native hardware by writing Objective-C plugins
Good documentation & large community

Cons

Overall performance of demos is poor
Contact list demos with huge amount of data just crash Safari (tested on hardware)
Building custom GUI is limited by HTML/CSS, there is a huge risk of ending up with unresponsive UI
A particular weakness of Phonegap is that it’s being migrated into Apache incubator & development/docs seem a bit messy
Adobe recently bought Phonegap & most probably their focus will shift towards Adobe’s AIR platform

Titanium (native approach)

Pros

Javascript, no need for Objective-C
Resulting app is Native!
Possibility to fully customize GUI
Produce HTML5 version of your application (works in browser!)
Ability to write cross-platform code for Android as well (also native!)
Titanium IDE (a free integrated development environment based on Eclipse, ex-Aptana)
Ability to make bindings to native hardware by writing Objective-C plugins
A shift towards iPhone development first (Android comes second) — which is adequate for current market shares of both
Community & company behind Titanium seems to be pretty responsive
Appcelerator recently raised 15 million series C funding, which means they are serious about the platformm & will
continue to improve it
Appcelerator is known to be largest third-party app publisher on both iOS and Android and is used by companies like NBC, Zipcar, eBay

Cons

Even though it’s Javascript, you’ll have to learn Titanium mobile API
Some developers had hard time debugging their apps & catching memory leaks
Documentation is somewhat messy & not as perfect if compared to Phonegap, however there is a positive improvement trend since the beginning of this year
A sort of platform lock-in, which means that in the long run it will be way too hard to switch to other mobile technology

Just to make it clear, Titanium does not produce fully native code from your javascript. It does not compile your code to ARM opcodes, what it does, in fact,
is binary javascript interpreter (Webkit’s JavascriptCore on iPhone, V8 on Android)

A note on hybrid approach

By hybrid approach I mean usage of HTML5/javascript with hooks into native iOS/Android API calls.

Of course, both Titanium and Phonegap fall into the category of “hybrid” but the key difference is implementation:
while Phonegap application runs inside browser, Titanium’s app runs inside javascript interpreter

Hybrid approach seemed like a real option, but in fact for custom/responsive GUI app you’ll inevitably need a lot of
code to support native UI. Hybrid works for small features like accessing accelerometer, camera, phone’s contacts, etc.

It seems that creating complex & responsive UI in HTML5 is not really possible, thus UX will be pretty
limited. Hybrid approach for the UI part is not a good fit from the development point of view:

Large amount of platform specific UI code (Obj-C) + javascript leads to technology fragmentation which is generally bad
a lot of quirks to connect native platform & javascript app leads to poor app design which will hurt you in long run

Conclusion

As you might guess, in the end it all comes to what type of application you’re going to build. If it’s mostly data-access, satellite mobile
application to your mainline app, than Titanium might be an option to consider. If you’re building something more similar to Path,
you’d better start right from hiring Objective-C folks.

And yet another warning, if you’re looking for cross-platform mobile application development, Titanium might not be a good choice.
Bad news, there’s no silver bullet here. Though, it does provide some mechanisms to do it well, but the dream of writing one single codebase
and ability to produce native apps for multiple platforms is very hard to achieve.

Native app for iPhone & Android (rich UX, more happy users)
Huge company behind Titanium, paid support which means you’ll not be left with your problems alone
Hundreds of successful apps already made with Titanium which fit Nimble API model (i.e. they are not standalone)

	Phonegap	Titanium
Language	Javascript	Javascript
Dev toolkit	poor	IDE, almost full integration
Custom GUI	Bound by browser’s CSS engine	Native capabilities
3-rd party native plugins	Yes	Yes, there is even marketplace
Community	Large	Large
Documentation	Good	Good
JS skills required	Web-dev is enough	Need to learn custom APIs
App performance	Mediocre	Good
Debugging	Easy	Hard
iPhone support	Yes	Yes, first-class
Android support	Yes	Yes, second-class

References

Preparing AdHoc distribution in Titanium Mobile

bananos — Tue, 14 Feb 2012 16:39:26 +0000

Appcelerator Titanium is a very promising technology which allows you to write native mobile applications in Javascript for both iPhone & Android.

After playing around a bit with Titanium Mobile, I decided to showcase technology strengths to our stakeholders, particularly by letting them use KitchenSink demo for iOS.

In order to distribute iOS application into real hardware you’ll have to pass through a bunch of Apple’s obstacles including:

Registering as an apple developer & getting valid certificate
Registering your app ID
Creating provisioning profile for specific iPhone UDIDs
Mess around with certificates/private keys to properly sign your application

It turns out that it’s not that easy to produce valid AdHoc distribution of your iOS app in Titanium, in this post I’m going to show you how to do it.

Before we start, make sure you have latest Xcode & Titanium SDK/Studio installed, as of the time of this writing Xcode version was 4.2.1, and Titanium mobile SDK was 1.8.1

For demo purposes we’ll use KitchenSink application provided as a showcase of Titanium mobile:

Clone & run KitchenSink in emulator

git clone https://github.com/appcelerator/KitchenSink.git

Open Titanium Studio and select File->Import->Import Existing Titanium Project

Configure project for iphone only

Run project on iOS simulator

At this point, Titanium will create an Xcode project for KitchenSink application. Wait for it to compile and run, then exit Simulator.

Open generated Xcode project

Run Xcode, and open KitchenSink.xcodeproj , Objective-C code could be found at kitchensink/build/iphone

Manage project schemas

In Xcode, open Product->Edit scheme:

For each schema (KitchenSink, KitchenSink-iPad, KitchenSink-universal) select Archive and make sure that
Build Configuration is set to debug

Configure code signing for build targets

For each target, select Build Settings -> Code Signing -> Code Signing Identity

Make sure to select valid provisioning identity that you previously created on Apple’s provisioning portal for a particular iOS device/application. In my case, it was called com.nimble.KitchenSink and provisioning profile was called KitchenSink_AdHoc. For identities to be recognized by Xcode you should have been installed Apple Developer’s certificate as well as your organization certificate into MacOS keychain.

Archive (produce final IPA file)

Select KitchenSink target, then from the main menu choose Product->Archive

Wait for Xcode to compile & archive. You may find resulting Archive in Xcode Organizer:

Click “Share” and provide the same identity which was used for Code Signing section above:

Save resulting IPA file along with corresponding KitchenSink_Adhoc.mobileprovision profile.

Install it on device

That’s the easiest part if you’ve done everything right. Just drag&drop mobileprovision & ipa files into iTunes and run sync with an actual iOS device.

If you missed something for code signing, during iTunes sync, you might encounter errors like this:

The executable was signed with invalid entitlements.
The entitlements specified in your application’s Code Signing Entitlements file do not match those specified in your provisioning profile.

Why GWT?

bananos — Mon, 10 Oct 2011 11:35:48 +0000

GWT is a set of tools which allows you to write javascript code in java language. That is, to avoid confusion, GWT does not have a lot in common with tons of existing Java technology stacks out there.

It is designed in a such way that backing your javascript application by server build on top of Java stack becomes a huge advantage. However, it does not mean that you can’t build GWT apps backed by PHP/Python/Ruby servers which might be more lightweight solution (both from technology & business points of view).

You can easily implement your data exchange using JSON or use ports for your favorite language of native GWTs RPC serialization mechanism.

One might be curious about GWT language choice, but If you’ll google a little bit about available Java projects & tools, the solution google folks made, becomes very obvious. Java as a language & technology proven to be successful over long period of time. By using Java in GWT you’re able to leverage good stuff from both worlds.

The bottom-line is as following: if you want to use GWT in your project there only few prerequisites:

Your devs are familiar with basic OOP principles & are strong in JavaScript for modern browsers
You are ready to pay an upfront cost of GWT/java lurning curve & overall initial slowness of development cycle

What are the main problems GWT was designed to solve

The problem of large-scale javascript development

Javascript as tool was not designed for writing huge applications, therefore it has some serious problems. Lack of tools, huge cost of maintenance, debugging issues, differences between Javascript engines, etc. (for more details see p.5)

The problem of building desktop-alike applications in browser (RIA)

From the very beginning you should clearly understand the difference between web site & web application, because some people confuse between these two.

A web site is a collection of related documents or information resources containing images, videos or other digital assets.

A web application is an application that is accessed over a network such as the Internet or an intranet.

What this means, is that you have to use completely different approach for developing complex web-apps. As a consequence, your web-app will be a more like a desktop GUI application from the architectural point of view.

If it’s more about GUI, than you’d better use somewhat time-tested, bulletproof architectures like MVC/MVP and look for their implementations in your technology stack. Otherwise, without having that kind of backbone, within a couple of months, blindly coding, you’ll end up with a pile of unmanageable crap.

The conclusion is as following:

It’s too risky to develop large javascript apps without special tools to manage complexity

What kind of tools we’re talking about? Simply put, those could be categorized as:

Debugging tools (Firebug, IE Dev tools, Webkit developer tools, etc.)
Packaging/optimization/obfuscation tools (YUI Compressor, Closure Compiler, JSMin, etc. )
Abstraction layers implemented as frameworks/libraries (jQuery, jQuery UI, YUI Library, SproutCore, Javascript MVC, Cappuccino, PersistenceJS, etc.)

Each of these tools focuses on a number of particular problems: crossbrowser issues(jQuery), building consistent GUI(jQueryUI), abstracting data layer(PersistenceJS), offering a full implementation stack of well-known architecture pattern (Sproutcore, Javascript MVC).

It’s up to you wether to use them or not, but in 2009, when we made evaluation(see spikes section p.7) none of mentioned solutions proven to meet our needs. Some of them were pretty buggy, others were not mature enough and some of them didn’t have a strong community.

As you probably see, there are tons of different ways on how to write your webapp in javascript, but

there’s no stable and consistent technology which is widely adopted for developing large-scale webapps

Should I use GWT or just stick with modern Javascript/HTML5 stack?

That is not the right question to ask. It’s not wether you should use GWT or not, but what kind of user experience do you want to deliver?

As I already mentioned, GWT will allow you to build feature-rich user interfaces and deliver desktop-alike experience to your end users: almost instant action response, rich interactions with complex business logic. GWT is not for creating web-sites, however your application will be run in browser. From this point of view, browser is just a way to deliver application to your customers.

Check this out, if you’re still in doubt by making your decision in favor of GWT:

Of course, no-one can prohibit you from building your RIA by means of modern HTML5 technologies, using plain JavaScript. But you must be aware that this will probably work only for tiny projects. Why?

Javascript flaws

As you might heard, Javascript was invented as a tool for scripting purposes by Netscape developers. The main goal of javascript was to write small programs which would enhance behavior of web pages.

Later on, it was adopted in other areas as a general tool for automation by system administrators, graphic artists, 3D modeling industry, etc.

An important thought here, is that javascript was not designed for writing real-world complex apps from the beginning:

No package mechanism
By default everything is in the global namespace
Only basic building blocks for OOP
Debugging might be a real nightmare
Lack of development tools
Javascript has its own flaws as a language
Scales pretty bad on a large number of developers

That said, you’ll inevitably need a special tooling to develop large apps in javascript

Pros & cons of GWT

Pros:

Mature technology stack backed by Java(OOP & strong typing is a huge win on large projects)
Debugging tools
By using Java, code maintenance & refactoring becomes really fun to do
Comprehensive, easily restyled widget library + GWT designer to build complex GUI interfaces
Security against JavaScript attacks
Logging
Accessibility
Internationalization
Data transport / management mechanisms
Crossbrowser
MVP framework
Unit testing framework
HTML5 support
Open-source
Tons of great libraries & frameworks (Mvp4g, GwtQuery)
Strong and active community, supported by Google
Deployment & optimization tools
Scales well by increasing number of developers

And one of the major advantages of GWT is:

At the end, it’s still javascript, which means that you can leverage all existing javascript/HTML5 technologies either by using plugins or by writing native javascript through JSNI

Cons:

Initial java-related overhead for setup/configuration (dev-mode, special plugins for modern browsers, etc.)
Speed of development is highly affected, you need top-notch hardware to comfortably write code
A lot of data abstraction layers are highly coupled with Java backends
Relatively high entrance threshold for new developers

You might want to check out feature comparisons matrix for different javascript frameworks as well.

Spike scenarios to examine different technology stacks

Data abstraction, packaging, event-binding spike. There should be a simple form with one textfield. As you type, 2 different widgets on the test page should update themselves with a data from textfield. Those 2 widgets should be controlled by independent business logic in separated packages(modules) of application.
Form validation spike. This will examine how easy it is to validate user data, including validation made on server-side.

By implementing first spike & evaluating your solution, ask yourself about units of functionality that will make up your future application:

How will those pieces be organized into units of code?
How will those pieces communicate with each other?
How will dependencies between components be expressed and managed while adhering to the principle of loose coupling?
How will components manifest themselves in the DOM? Do they need to?
How will we persist data across URL and page loads?
How will we manage communication with the server?
How will we make sure users only see the data they’re allowed to see?

There is no spike whatsoever which can show you strengths of GWT in managing large-scale apps right away. This is obvious — until you’re going to play with some trivial samples, it might look as an unnecessary technology overhead. But it greatly pays off in the long run. Our 2+ years of frontend development proved GWT to be the right choice.

You might be interested to investigate why others are using GWT & what are the type of projects being built.

Hidden cost of transition from GWT to javascript

What if, after huge investment in GWT you decided to switch back to Javascript?

That is a pretty serious question from the business perspective. Remember, that we’re talking about apps here, not web-pages. This means, that your existing GWT application already carries a lot of complex business logic, it’s not a simple presentation layer as one might think.

Keeping this in mind, you’ll probably end up with:

Complete rewrite of application (you’d better want to spent this time for new valued features to compete in your market, don’t you?)
Hiring pretty expensive javascript-gurus (you don’t want those numerous folks writing random jQuery snippets to develop your complex business-logic app, do you?)

From these points of view, transition doesn’t seem to be a good idea, even though you might find out that GWT resources are pretty expensive. The trick why those folks are so expensive is that they specialize a lot on GWT backend stacks, which is an enterprise class java technologies. Instead, if your back-end stack is non-java, you could happily hire senior javascript guys & easily convert them into GWT devs. Our 2+ years experience showed that it works pretty well — monthly learning curve and your dev-team is able to use all of GWT strengths.

How to leverage existing Javascript solutions?

Javascript is so widespread today and there are tons of cool stuff already built. How is it possible to grab some piece from that tasteful cake?

You’re probably looking at these:

Cost of javascript resources is pretty attractive
Re-use of existing open-source javascript code

But wait, is GWT really java? Nope. At the end it’s the same old javascript which is used everywhere. What this means for you as business, is that you could still leverage any javascript solutions out there.

In the case of existing GWT application, the solution seems to be pretty obvious. Core GUI front-end may be maintained using GWT & third-party extensions can be easily built using plain javascript. However, this does bring a few of technical problems:

There is a need to invent a well designed interface for integration of GWT Core app and third-party javascript
There is a problem of css-styling of core app by means of third-party extensions

Those are a real technical problems, which you just need to spent some time to resolve. This way, you’ll continue to happily use GWT for the “core” stuff & leverage javascript extensions which might add a huge value to your application.

Python vs. PHP duck typing

bananos — Fri, 12 Aug 2011 12:44:38 +0000

Having previous experience in writing PHP code could potentially lead you to hours of debugging in Python.
Consider the following simple example in PHP:

PHP


$s = "0";
$my = (bool) $s;

var_dump($s);
var_dump($my)

And the output would be:


string(1) "0"
bool(false)

Which is pretty expectable behavior in PHP, since it first casts string to int, and than to boolean.

Consider the same example in Python:

Python


s = "0"
my = bool(s)
print type(s), s
print type(my), my

And the output would be:



 0
 True

We’re not going to debate here wether this is good or bad, but you should be aware such things before they’ll steal hours of your time

To achieve the same behavior as in PHP, you’ll need to add explicit integer typecast:


s = "0"
my = bool(int(s))
print type(s), s
print type(my), my

And the output would be:



 0
 False

Syntax highlighting for Nano on Mac OS X

bananos — Fri, 22 Jul 2011 14:12:07 +0000

Nano is one of my favorite text editors, for a couple of reasons

It’s very simple (as campared to Vim, which I was not able to quickly grasp on)
It’s available by default on almost any *nix distro out there
It has syntax highlighting (always turned off by default for some reason)

It this post I’ll show you how to install latest Nano to your Mac OS X environment with syntax highlighting turned on for most of programming&markup languages.

For managing my OSX packages I’m using brew, but unfortunately it does not have nano formula yet. Built-in version of nano is pretty outdated (2.0.6), and there is no binaries available exclusively for mac. So we’re going to do a classic configure-make-install dance.

Step1. Download nano sources

wget http://www.nano-editor.org/dist/v2.2/nano-2.2.6.tar.gz

Step2. Download syntax highlighting configs for nano

wget http://webapp.org.ua/wp-content/uploads/2011/07/nanorc.tar

These are compiled from different sources over the internet with some small mods made by myself.

Step3. Compile & install


tar -xvvf nano-2.2.6.tar.gz
cd nano-2.2.6
sudo ./configure --prefix=/usr/
sudo make install
cd /
sudo tar -xvvf nanorc.tar

At this point you’re done. Latest nano with syntax highlighting will be installed at your system paths.

Compiling uWSGI from sources

bananos — Fri, 03 Jun 2011 08:33:47 +0000

As you probably know, uWSGI is a fast application container for serving your webapps. It is often used with Django against nginx webserver. Written in pure C & having a lot of other cool features.

It seems that installation of uWSGI is pretty simple according to official docs but it isn’t.

If you want to deploy your Django app on a brand new linux box (VPS/Amazon EC2), using buildout without a lot of dependencies things become more complicated.

Here is how you can tune up your uWSGI:

wget http://projects.unbit.it/downloads/uwsgi-0.9.7.2.tar.gz
tar -xvvf  uwsgi-0.9.7.2.tar.gz
cd uwsgi-0.9.7.2

If you’ll simply run make as this point, you’ll experience something like this:

 *** libxml2 headers unavailable. uWSGI build is interrupted. You have to install libxml2 development package or use libexpat or disable XML

Since uWSGI needs libxml2 for reading XML configs, which is only one of the available configuration options (command line, environment variables, XML, .ini, yaml files and via LDAP.), we’re going to drop this dependency (I’m more comfortable with command-line style)

It’s not obvious how to drop libxml2 dependency, because there’s no configure & usual C-style make. Instead, build is fully controlled by uwsgiconfig.py script, which you can grok for a while to get a grasp on how the whole process works.

Build options are stored in buildconf/ folder:

root@devbox:~/server/thirdparty/uwsgi-0.9.7.2# ls -l buildconf/
total 64
-rw-r--r-- 1 root root  50 Apr  6 10:50 core.ini
-rw-r--r-- 1 root root 497 Apr  6 10:50 default.ini
-rw-r--r-- 1 root root  54 Apr  6 10:50 erlang.ini
-rw-r--r-- 1 root root  49 Apr  6 10:50 lua.ini
-rw-r--r-- 1 root root  40 Apr  6 10:50 luap.ini
-rw-r--r-- 1 root root 485 Apr  6 10:50 modular.ini
-rw-r--r-- 1 root root  78 Apr  6 10:50 package.ini
-rw-r--r-- 1 root root  50 Apr  6 10:50 psgi.ini
-rw-r--r-- 1 root root  60 Apr  6 10:50 pyerl.ini
-rw-r--r-- 1 root root  86 Apr  6 10:50 pylua.ini
-rw-r--r-- 1 root root 462 Apr  6 10:50 pyrb.ini
-rw-r--r-- 1 root root  69 Apr  6 10:50 rack.ini
-rw-r--r-- 1 root root 449 Apr  6 10:50 rackp.ini
-rw-r--r-- 1 root root 377 Apr  6 10:50 unbit.ini
-rw-r--r-- 1 root root 108 Apr  6 10:50 unbitstaff.ini

For our customized build we’ll create a new build config, called myproject.ini:

root@devbox:~/server/thirdparty/uwsgi-0.9.7.2/buildconf# cp default.ini myproject.ini

He’re how internals should look like if you want to turn off libxml2 support:


[uwsgi]
xml = false
ini = true
yaml = true
snmp = true
sctp = false
spooler = true
embedded = true
udp = true
multicast = true
threading = true
sendfile = true
minterpreters = true
async = true
evdis = false
ldap = auto
pcre = auto
debug = false
unbit = false
; xml_implementation = libxml2
plugins =
bin_name = uwsgi
plugin_dir = .
embedded_plugins = python, ping, nagios, rpc, fastrouter, http, ugreen

locking = auto
event = auto
timer = auto
filemonitor = auto

[python]
paste = true
web3 = true

At this point you’re ready to type

 python uwsgiconfig.py --build myproject

In theory this should build you a brand-new uWSGI, but only in theory. Probably, what you’ll get by doing above command will be:

using profile: buildconf/myproject.ini
*** uWSGI compiling server core ***
gcc -pthread -c utils.c -o utils.o -O2 -Wall -Werror -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64  -fno-strict-aliasing -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -DUWSGI_BUILD_DATE="\"03 June 2011 09:10:42\"" -DUWSGI_LOCK_USE_MUTEX -DUWSGI_EVENT_USE_EPOLL -DUWSGI_EVENT_TIMER_USE_TIMERFD -DUWSGI_EVENT_FILEMONITOR_USE_INOTIFY
gcc -pthread -c protocol.c -o protocol.o -O2 -Wall -Werror -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64  -fno-strict-aliasing -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -DUWSGI_BUILD_DATE="\"03 June 2011 09:10:42\"" -DUWSGI_LOCK_USE_MUTEX -DUWSGI_EVENT_USE_EPOLL -DUWSGI_EVENT_TIMER_USE_TIMERFD -DUWSGI_EVENT_FILEMONITOR_USE_INOTIFY
gcc -pthread -c socket.c -o socket.o -O2 -Wall -Werror -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64  -fno-strict-aliasing -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -DUWSGI_BUILD_DATE="\"03 June 2011 09:10:42\"" -DUWSGI_LOCK_USE_MUTEX -DUWSGI_EVENT_USE_EPOLL -DUWSGI_EVENT_TIMER_USE_TIMERFD -DUWSGI_EVENT_FILEMONITOR_USE_INOTIFY
gcc -pthread -c logging.c -o logging.o -O2 -Wall -Werror -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64  -fno-strict-aliasing -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -DUWSGI_BUILD_DATE="\"03 June 2011 09:10:42\"" -DUWSGI_LOCK_USE_MUTEX -DUWSGI_EVENT_USE_EPOLL -DUWSGI_EVENT_TIMER_USE_TIMERFD -DUWSGI_EVENT_FILEMONITOR_USE_INOTIFY
gcc -pthread -c master.c -o master.o -O2 -Wall -Werror -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64  -fno-strict-aliasing -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -DUWSGI_BUILD_DATE="\"03 June 2011 09:10:42\"" -DUWSGI_LOCK_USE_MUTEX -DUWSGI_EVENT_USE_EPOLL -DUWSGI_EVENT_TIMER_USE_TIMERFD -DUWSGI_EVENT_FILEMONITOR_USE_INOTIFY
cc1: warnings being treated as errors
master.c: In function ‘uwsgi_subscribe’:
master.c:74: error: implicit declaration of function ‘send_udp_message’

and your build will miserably fail

To fix that, you need to edit uwsgiconfig.py, find following line:

self.cflags = ['-O2', '-Wall', '-Werror', '-D_LARGEFILE_SOURCE', '-D_FILE_OFFSET_BITS=64'] + os.environ.get("CFLAGS", "").split()

and remove -Werror

In case you want to do it automatically by your deployment scripts, here is a small sed snippet:

sed  -ie "s#'-Werror',##g" uwsgiconfig.py

After patching uwsgiconfig.py you’re ready to rock&roll:

  python uwsgiconfig.py --build myproject

Weird rocketry » dev

OpenWRT Chaos Calmer 15.05 on Dlink DIR-320 with CDMA modem

Install OpenWRT using Image builder for brcm47xx/legacy

Firmware Flashing

Setting up extroot

Installation of package manager (opkg)

Make sure internet works

LuCI installation

3g/CDMA modem configuration

Measuring signal strength

TL;DR;

Links

Running php 5.2 fpm on ARM processor

Compiling from sources

PHP-FPM patch

libxml2 patching

Atomic types

TL;DR

Sources

CloudWatch custom metrics

Install AWS CLI

Choose a metric

Choose metric name

Periodic script execution

Setup alarms

Gotchas

Links

Intellij IDEA and Go plugin

Install IDEA community edition

Grab IDEA sources!

Get Intellij IDEA community sources

Grab Go plugin sources

Install plugin

Install Go SDK

TL;DR

Phonegap vs. Titanium

Phonegap (hybrid approach)

Pros

Cons

Titanium (native approach)

Pros

Cons

A note on hybrid approach

Conclusion

References

Preparing AdHoc distribution in Titanium Mobile

Clone & run KitchenSink in emulator

Configure project for iphone only

Run project on iOS simulator

Open generated Xcode project

Manage project schemas

Configure code signing for build targets

Archive (produce final IPA file)

Install it on device

Why GWT?

What are the main problems GWT was designed to solve

The problem of large-scale javascript development

The problem of building desktop-alike applications in browser (RIA)

Should I use GWT or just stick with modern Javascript/HTML5 stack?

Javascript flaws

Pros & cons of GWT

Pros:

Cons:

Spike scenarios to examine different technology stacks

Hidden cost of transition from GWT to javascript

How to leverage existing Javascript solutions?

Python vs. PHP duck typing

PHP

Python

Syntax highlighting for Nano on Mac OS X

Step1. Download nano sources

Step2. Download syntax highlighting configs for nano

Step3. Compile & install

Compiling uWSGI from sources