All posts by cuquejo

SysAdmin/DevOps Professional with strong Linux focus, experience with design and support of high availability webscale infrastructures and resilient database deployments. Deep understanding of Linux, Cloud, Information Security and outsource support. If you need to send me an encrypted mail. Please import my GPG key: gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys 63B19B1C52B7AC98033EAC670F6A2073E0EE5DC5 If this blog helps you, you could buy me a coffee: BTC: 33NMM2NAWYqpTwzDTEhFFKpikKw5aMjNkR LTC: M8PdbscG4Ad3i7VVQgr5o4jG8MMTrbzR6B ETH: 0x00434CC65Ec092d52c654ac8F1B2D8EDB9bEB6da

Kafka tools – dump / debug a topic

How to use kafkacat + ssl

BROKERS="{myBrokerAddress}:{PORT}"
tpc=myTopic
kafkacat -b "$BROKERS" \
  -X security.protocol=SSL \
  -X ssl.key.location=/etc/ssl/kafka-client.key \
  -X ssl.certificate.location=/etc/ssl/kafka-client.crt \
  -X ssl.ca.location=/etc/ssl/kafka-client.caroot.crt -J \
  -C -o beginning \
  -t "$tpc"

Kafkacat Options

General options:
  -C | -P | -L | -Q  Mode: Consume, Produce, Metadata List, Query mode
  -G       Mode: High-level KafkaConsumer (Kafka 0.9 balanced consumer groups)
                     Expects a list of topics to subscribe to
  -t          Topic to consume from, produce to, or list
  -p      Partition
  -b <brokers,..>    Bootstrap broker(s) (host[:port])
  -D          Message delimiter character:
                     a-z.. | \r | \n | \t | \xNN
                     Default: \n
  -E                 Do not exit on non fatal error
  -K          Key delimiter (same format as -D)
  -c            Limit message count
  -X list            List available librdkafka configuration properties
  -X prop=val        Set librdkafka configuration property.
                     Properties prefixed with "topic." are
                     applied as topic properties.
  -X dump            Dump configuration and exit.
  -d <dbg1,...>      Enable librdkafka debugging:
                     all,generic,broker,topic,metadata,feature,queue,msg,protocol,cgrp,security,fetch,interceptor,plugin,consumer,admin,eos
  -q                 Be quiet (verbosity set to 0)
  -v                 Increase verbosity
  -V                 Print version
  -h                 Print usage help

Producer options:
  -z snappy|gzip     Message compression. Default: none
  -p -1              Use random partitioner
  -D          Delimiter to split input into messages
  -K          Delimiter to split input key and message
  -l                 Send messages from a file separated by
                     delimiter, as with stdin.
                     (only one file allowed)
  -T                 Output sent messages to stdout, acting like tee.
  -c            Exit after producing this number of messages
  -Z                 Send empty messages as NULL messages
  file1 file2..      Read messages from files.
                     With -l, only one file permitted.
                     Otherwise, the entire file contents will
                     be sent as one single message.

Consumer options:
  -o         Offset to start consuming from:
                     beginning | end | stored |
                       (absolute offset) |
                     - (relative offset from end)
  -e                 Exit successfully when last message received
  -f          Output formatting string, see below.
                     Takes precedence over -D and -K.
  -J                 Output with JSON envelope
  -D          Delimiter to separate messages on output
  -K          Print message keys prefixing the message
                     with specified delimiter.
  -O                 Print message offset using -K delimiter
  -c            Exit after consuming this number of messages
  -Z                 Print NULL messages and keys as "NULL"(instead of empty)
  -u                 Unbuffered output

Metadata options (-L):
  -t          Topic to query (optional)

Query options (-Q):
  -t :
:    Get offset for topic ,
                     partition 
, timestamp .
                     Timestamp is the number of milliseconds
                     since epoch UTC.
                     Requires broker >= 0.10.0.0 and librdkafka >= 0.9.3.
                     Multiple -t .. are allowed but a partition
                     must only occur once.

Format string tokens:
  %s                 Message payload
  %S                 Message payload length (or -1 for NULL)
  %R                 Message payload length (or -1 for NULL) serialized
                     as a binary big endian 32-bit signed integer
  %k                 Message key
  %K                 Message key length (or -1 for NULL)
  %T                 Message timestamp (milliseconds since epoch UTC)
  %t                 Topic
  %p                 Partition
  %o                 Message offset
  \n \r \t           Newlines, tab
  \xXX \xNNN         Any ASCII character
 Example:
  -f 'Topic %t [%p] at offset %o: key %k: %s\n'


Consumer mode (writes messages to stdout):
  kafkacat -b  -t  -p 
 or:
  kafkacat -C -b ...

High-level KafkaConsumer mode:
  kafkacat -b  -G  topic1 top2 ^aregex\d+

Producer mode (reads messages from stdin):
  ... | kafkacat -b  -t  -p 
 or:
  kafkacat -P -b ...

Metadata listing:
  kafkacat -L -b  [-t ]

Query offset by timestamp:
  kafkacat -Q -b broker -t ::</dbg1,...></brokers,..>

Debug and Work with Kafka REST API using Insomnia

Install Insomnia and configure

  • Install Insomnia app
  • Download my Kafka REST API Workspace (here).
  • Then import this Workspace file to you Insomnia:
    Workspace -> Import/Export
  • Click on you dev  -> Manage Environments and update the base_url to point to your local Kafka REST URL
    {
      "base_url": "http://EXAMPLE_KAFKA_REST:8082"
    }

How to Produce data and create a consumer for Binary Topics?

  • Open the Binary Topic Folder:
  • Click on Producer and replace the topic name here:
  • Replace the Json content to wherever you want and click in send, if all works fine you will get an HTTP 200 OK response.
  • Create a consumer by clicking on “1Add Consumer group-id”, and replace the consumer name and group:
    Click send and if OK you will see a “200 OK” and the base_uri for your new consumer.
  • Then Subscribe this group to the topic that you want to consume, by clicking on “2Subscribe this group-id to a topic”, and replace the consumer name and group in the URL and replace the topic name on the Body message:

    Press send and if you receive a “204 No Content response” all good, the consumer is created and listening to your topic.
  • Finally, you could Consume the data by selecting “3Consume” and replace the consumer and group like done in the previous step.

Git Tips and Tricks

!!! WORK IN PROGRESS !!!

Some useful commands

Setup git to use vscode has the default editor

Run this command and then add this lines:

$ git config --global -e
[core]
	editor = code --wait

[diff]
	tool = default-difftool

[difftool "default-difftool"]
  cmd = code --wait --diff $LOCAL $REMOTE

[difftool]
	prompt = false

[merge]
	tool = code

[merge "tool"]
	cmd = "code --wait $MERGED"
	prompt = false
	keepbackup = false

How to compare (diff) files from two different branches

git difftool mybranch anotherbranch -- myfile.txt

or

git diff branch1:file branch2:file

commit –amend

Edit a commit before push it, example:

$ touch test.txt
$ git add test.txt
$ git commit -m "Test file"
[master 964fa35] Test file
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 test.txt
$ git log --oneline
964fa35 (HEAD -> master)Test file
$ touch test2.txt
$ git add test2.txt 
$ git commit --amend -m "Test and Test2 file"
$ git log --oneline
414a06e (HEAD -> master) test and test2

git log

git log shows the current HEAD and its ancestry. That is, it prints the commit HEAD points to, then its parent, its parent, and so on.

$ git log --oneline
$ git log HEAD@{1.hour.ago}
$ git log HEAD@{1.week.ago}

git reflog

git reflog doesn’t traverse HEAD’s ancestry at all. The reflog is an ordered list of the commits that HEAD has pointed to: it’s undo history for your repo. The reflog isn’t part of the repo itself (it’s stored separately to the commits themselves) and isn’t included in pushes, fetches or clones; it’s purely local.

$ git reflog HEAD@{1.hour.ago} --oneline

How to clear unreachable commits

$ git reflog expire --expire-unreachable=now --all
$ git gc --prune=now

Install PHP 7.3 + xdebug on MacOS Mojave (with homebrew)

First, you need to have homebrew installed, if you don’t have it please go to https://brew.sh/ and follow the installation steps.

Remove any old version of php:

brew list | grep php | awk '{ print "brew uninstall --force "$1}' | bash

To be sure, manually remove the php cellar folder:

rm -rf /usr/local/Cellar/php

Then clean any php tap (extra repo):

brew untap homebrew/php

Cleanup and Update your brew:

brew cleanup
brew update

Install the latest version of php:

brew install php

Clean up old xdebug:

pecl uninstall xdebug

Install the latest version of xdebug-beta (The xdebug is not compatible with 7.3 so you need to use the xdebug-beta):

pecl install xdebug-beta

The last step, edit the  /usr/local/etc/php/7.3/php.ini and add these lines at the bottom of this file:

zend_extension=xdebug.so
[XDebug]
xdebug.remote_enable=1
xdebug.remote_autostart=1
xdebug.remote_handler=dbgp
xdebug.remote_mode=req
xdebug.remote_host=127.0.0.1
xdebug.remote_port=9000

Gcloud Command Reference

Gcloud Command Reference

Some trick configuration

To set the project property in the core section, run:

$ gcloud config set project myProject

To set the zone property in the compute section, run:

$ gcloud config set compute/zone us-central1-a

To disable prompting for scripting, run:

$ gcloud config set disable_prompts true

To set a proxy with the appropriate type, and specify the address and port on which to reach it, run:

$ gcloud config set proxy/type http
$ gcloud config set proxy/address 1.234.56.78
$ gcloud config set proxy/port 8080

Show your running instances

$ gcloud compute instances list
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
test3-jump-001 us-central1-a f1-micro 10.200.1.2 200.200.200.17 RUNNING
test3-node-001 us-central1-a n1-standard-1 10.200.252.6 RUNNING
test3-node-002 us-central1-a n1-standard-1 10.200.252.2 RUNNING
test3-node-003 us-central1-a n1-standard-1 10.200.252.3 RUNNING
test3-node-004 us-central1-a n1-standard-1 10.200.252.4 RUNNING
test3-node-005 us-central1-a n1-standard-1 10.200.252.5 RUNNING

Working with Buckets

To set create a new bucket, run:

$ gsutil mb gs://mySuperNewBucket

To enable versioning on it, run:

$ gsutil versioning set on gs://mySuperNewBucket

To remove all files on the bucket

$ gsutil rm -r gs://mySuperNewBucket

To remove the bucket (needs to be empty)

$ gsutil rb gs://mySuperNewBucket

Listing zones 

To show all available zones for your account, run:

$ gcloud compute zones list
NAME REGION STATUS NEXT_MAINTENANCE TURNDOWN_DATE
us-east1-b us-east1 UP
us-east1-c us-east1 UP
us-east1-d us-east1 UP
us-east4-c us-east4 UP
us-east4-b us-east4 UP
us-east4-a us-east4 UP
us-central1-c us-central1 UP
us-central1-a us-central1 UP

Listing machine-types and show CPU memory sizes

You could filter per region including a filter on it, example:

$ gcloud compute machine-types list --filter="zone ~ ^us-central1-a"
NAME ZONE CPUS MEMORY_GB DEPRECATED
f1-micro us-central1-a 1 0.60
g1-small us-central1-a 1 1.70
n1-highcpu-16 us-central1-a 16 14.40
n1-highcpu-2 us-central1-a 2 1.80
n1-highcpu-32 us-central1-a 32 28.80
n1-highcpu-4 us-central1-a 4 3.60
n1-highcpu-64 us-central1-a 64 57.60
n1-highcpu-8 us-central1-a 8 7.20
n1-highcpu-96 us-central1-a 96 86.40
n1-highmem-16 us-central1-a 16 104.00
n1-highmem-2 us-central1-a 2 13.00
n1-highmem-32 us-central1-a 32 208.00
n1-highmem-4 us-central1-a 4 26.00
n1-highmem-64 us-central1-a 64 416.00
n1-highmem-8 us-central1-a 8 52.00
n1-highmem-96 us-central1-a 96 624.00
n1-standard-1 us-central1-a 1 3.75
n1-standard-16 us-central1-a 16 60.00
n1-standard-2 us-central1-a 2 7.50
n1-standard-32 us-central1-a 32 120.00
n1-standard-4 us-central1-a 4 15.00
n1-standard-64 us-central1-a 64 240.00
n1-standard-8 us-central1-a 8 30.00
n1-standard-96 us-central1-a 96 360.00

Connect to a compute instance using the gcloud credentials

To access one instance through your gcloud credentials, run:

$ gcloud compute --project "myCoolProjectName" ssh --zone "us-east1-b" "myCoolInstanceName"

Find how much memory apache threaded MPM is using

The Worker MPM turns Apache into a multi-process, multi-threaded web server. Unlike Prefork, each child process under Worker can have multiple threads.

If you use top or ps you will get something like this:
top
And sometimes is hard to find how many resources this service is using.

If you want to have the memory usage grouped by process, we could use the atop tool and then press ‘m’ (memory usage) and ‘p’ (group by process name). Example:

atop

This will work with any multi-process, multi-threaded service 🙂

Correct Terminal prompt wrapping

Sometimes I get a wrong wrapping when using terminal via ssh and I find out is mostly to do with the size of the window assumed by the terminal is not the same as your actual window size.
E.g.:
Wrong Wrapping

To solve this:

$ shopt checkwinsize
And check if the checkwinsize is eanbled, if not:

$ shopt -s checkwinsize
Then just run another command or resize your window size.

You could put this command on your ~/.bashrc or /etc/bashrc

VMWare ESXi command-line

List all vms

vim-cmd vmsvc/getallvms

Check the power state of the virtual machine with the command:

vim-cmd vmsvc/power.getstate <vmid>

Power-on the virtual machine with the command:

vim-cmd vmsvc/power.on <vmid>

Retrieving Networking Information

esxcli network ip interface list
esxcli network ip interface ipv4 get -i vmk<X>

Register a VMX file to a VM

vim-cmd solo/registervm /vmfs/volumes/datastore_name/VM_directory/VM_name.vmx

Reply Question after register a VM from a VMX file

vim-cmd vmsvc/message <vmid>
vim-cmd vmsvc/message <vmid> <question_id> <answer_id>

vSwitch

How to Configure VMware vSwitch from ESX / ESXi Command Line

Terraform Tips

This is just a brain dump to myself 🙂

I’m moving some functions from Ansible to Terraform and this is some basic commands:

How to find the CentOS 7 official AMI Image

aws ec2 describe-images \
    --owners 'aws-marketplace' \
    --filters 'Name=product-code,Values=aw0evgkw8e5c1q413zgy5pjce' \
    --query 'sort_by(Images, &CreationDate)[-1].[ImageId]' \
    --region 'eu-central-1' \
    --output 'text'

How to find the Ubuntu official Image

Ubuntu AWS ami Locator: https://cloud-images.ubuntu.com/locator/ec2/

AWS Marketplace (Filtered by SO and Free): https://aws.amazon.com/marketplace/

instance.tf

terraform apply # Will create all machines on instance.tf
terraform destroy # Destroy managed infrastructure
terraform plan # Will show but not apply
terraform plan -out exampletest.out #
terraform appy exampletest.out #

terraform import aws_instance.example i-abcd1234

This command locates the AWS instance with ID i-abcd1234 and attaches its existing settings, as described by the EC2 API, to the name aws_instance.example in the Terraform state.

Variables how it works

HortonWorks Hive enable auth using Ambari users

HortonWorks Hive enable auth using Ambari users

configure the admin users on Ambari

Access -> Hive -> Configs -> Advanced -> Custom hive-site -> Add Property
hive.users.in.admin.role=hive,admin

Save and reload hive.

Connect to the SSH of you hadoop cluster then run beeline and connect using hive and password hive command:

$ beeline
Beeline version 1.2.1000.2.6.1.0-129 by Apache Hive
beeline> !connect jdbc:hive2://hadoop-2:10000 hive hive  org.apache.hive.jdbc.HiveDriver
Connecting to jdbc:hive2://hadoop-2:10000
Connected to: Apache Hive (version 1.2.1000.2.6.1.0-129)
Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129)
Transaction isolation: TRANSACTION_REPEATABLE_READ

Set the admin role to hive user

0: jdbc:hive2://hadoop-2:10000> SET ROLE ADMIN;
No rows affected (2.192 seconds)

Create basic roles (*Optional*)

Create 2 roles one for read-only users and another one for read-write users;

1: jdbc:hive2://hadoop-2:10000> create role RWUSER;
No rows affected (0.229 seconds)
1: jdbc:hive2://hadoop-2:10000> create role ROUSER;
No rows affected (0.221 seconds)
1: jdbc:hive2://hadoop-2:10000> ALTER DATABASE DEFAULT SET OWNER ROLE RWUSER;

Enable Security Authorization

Go to Hive -> Config -> Settings -> Choose Authorization -> SQLStdAuth.

Save and Reload Hive