[lukemarsden]

DO NOT MERGE

This is a WIP.

This PR introduces:

• a new "quickstart" on how to use kubeadm to easily install Kubernetes
• a new "add-ons" page, intended to be a user-friendly central place for folks to go to look at the list of add-ons they can install on their cluster

Currently I've only added Weaveworks add-ons to the latter, but we should encourage other add-ons to be added to this list.

---

This change is

[lavalamp]

please squash this :)

[lukemarsden]

@lavalamp squashed :)

[caseydavenport]

Taking a first pass at this.

I didn't manage to get the ubuntu instructions to work - the provided install for kubelet-kubeadm didn't work. Not sure if it is meant to yet. Also some minor typos.

Also have a few concerns regarding the addons doc - mostly its relationship with the existing home for Kubernetes addons and the verbosity.

[caseydavenport]

I think it's a bit confusing that these are called "addons", given there are a bunch of addons that already exist in the cluster/addons directory that have different semantics than what is described here. How does this relate to the ones here? https://github.com/kubernetes/kubernetes/tree/master/cluster/addons

I think we either want to document the existing addons in that directory / the relationship this document has to the things in that directory, or rename the things in this document to something else (e.g "extensions", whatever).

[lukemarsden]

I am thinking what makes most sense is to split the definition of add-ons into "built-in add-ons" and "3rd party add-ons". The former will obey the semantics in the cluster addons README and the latter will be "just kubectl apply -f it" style external add-ons like Weave Net. How does that sound @caseydavenport?

[caseydavenport]

Hmm, I think there is some overlap between this comment and the comments below.

The best thing in the short term is probably to:

• keep calling all of these things addons, but not mandating an installation method. Each addon should document its own installation procedure. This keeps the docs leaner and easier to maintain, since the documentation for each addon lives with that addon, and some addons will be more / less complex than others.
• We can link to each of the existing /cluster/addons/X addons in this doc.
• As the old /cluster directory is torn down, those existing addons will either find new homes or go away.

WDPT?

[jbeda]

I'd love to see all addons move to being kubectl apply -f. It'll be much easier to install and manage. They were done the way they were done because we didn't have daemonsets at the time, etc.

[lukemarsden]

I'll have a go at wordsmithing this to get a balance between simplicity and explaining to users that we're transitioning from built-in addons to "self-hosted" ones.

[lukemarsden]

I have attempted to simplify the add-ons page per this (and other) discussion: https://lukemarsden.github.io/docs/admin/addons/

[caseydavenport]

See my comment here: https://github.com/kubernetes/kubernetes.github.io/pull/1265/files#r80270865

To summarize, I think the clarification re: addons vs the "/cluster/addons" directory is much better! I still think we could improve the rest of the page so that it comes across less like marketing.

[caseydavenport]

I think this line and link should be moved up and combined with the above "Several projects provide Kubernetes pod networks".

[philips]

+1 I love all of the work y'all are doing. But please lets centralize the docs to a single list and then link off to the external docs for installation. These docs are going to sprawl and break if we don't focus it to a single location.

[lukemarsden]

Sure, will make this change.

[lukemarsden]

I've moved the "You can see a complete list of available network add-ons" line up, and then repeated it again below.

[caseydavenport]

I think this document will get big and cumbersome if we add similar levels of detail to each arbitrary cluster extension that the community writes.

This doc should maintain a simple list, with links to the community-supported documentation/home for each item.

e.g

## Ingress Controllers

* [nginx](https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx)
* . . .

## Networking

* [Weave Net](https://github.com/weaveworks/weave-kube)
* . . .

## Cluster Visualization

* [Kubernetes Dashboard]( . . . )
* [Weave Scope]( . . . )
* . . .

[luxas]

Yes, I think that we can be a little less verbose here and instead document all addons listed in cluster/addons plus dashboard, ingress, etc.

[cdrage]

I'm assuming that alternative networking would be documented here as well in the future? (ex. when Canal combines weave + flannel or 1.0.0 of Flannel)

[caseydavenport]

@cdrage Yes, I have a PR open against this branch to do that very thing!

https://github.com/lukemarsden/lukemarsden.github.io/pull/1

[lukemarsden]

+1

I think we do need a 1 sentence description of each add-on though otherwise users have to click all the links to know what they're looking at. Keeping it to 1 sentence and linking to more verbose docs at the appropriate URL totally makes sense though.

[lukemarsden]

I also think eventually a table view might be good for this. It'll force us to keep the one sentence descriptions short. I would like to keep the kubectl apply -f copyable commands though to demonstrate how easy it is to install these add-ons to users.

[caseydavenport]

I really don't think the kubectl apply -f commands should be documented here. This needs to be just a collection of links, with any documentation living elsewhere.

A short description (less than one line) is fine, but that should be it. Installation instructions should live with the addon documentation.

[lukemarsden]

OK, I've removed the kubectl apply -f instructions from the addons page.

[caseydavenport]

s/containers/Pods ?

s/workload/workloads

[lukemarsden]

fixed, thanks!

[caseydavenport]

s/eachother/each other

[lukemarsden]

fixed, thanks!

[caseydavenport]

This line is in the future tense, but the user has already used the NodePort in the past :)

[lukemarsden]

Will fix

[lukemarsden]

fixed

[caseydavenport]

Adding HA support to the kubeadm tool is still a work-in-progress (Kubernetes supports it in the general case)

[lukemarsden]

fixed, thanks!

[caseydavenport]

s/will have/has

Looks like this section was intended to be at the top of the doc.

[lukemarsden]

yep

[lukemarsden]

fixed

[caseydavenport]

This is a WIP and the instructions do not yet work.

D'oh, I should read things better.

[luxas]

I don't think you meant to change this

[lukemarsden]

Will revert before I put this up for final review

[luxas]

Yes, I think that we can be a little less verbose here and instead document all addons listed in cluster/addons plus dashboard, ingress, etc.

[luxas]

This isn't great. If you strongly want this, tag commit weaveworks-experiments/weave-npc@40f5461 with a version like v0.1.0 or v1.0.1

[jbeda]

I'd love to see instructions like this move to a weave owned page. It'll be easier to keep up to date as weave changes and will scale better as we document more types of add-ons.

If we have community supported add-ons (that are part of the k8s project) perhaps we should create new pages for those so that they keep this page clean?

[lukemarsden]

Yeah, this was just a temporary placeholder URL. weave-npc is going to be merged into weave-kube so I will kill this section entirely.

[lukemarsden]

fixed

[luxas]

Install the kubeadm package on all machines

[jbeda]

Mention prerequisites? You'll need to have docker installed on each machine starting out.

[cdrage]

install what exactly? (kubectl / docker / rkt / kubeadm?)

[lukemarsden]

This tl;dr is not meant to be followed, it's meant to give a sense of how easy the process is to encourage people to continue. Prerequisites are mentioned below.

[lukemarsden]

Also, you don't need to have docker installed on each machine before starting out.

[lukemarsden]

I've killed the tl;dr section, it was basically a repetition of the objectives section and was causing confusion.

[luxas]

...install a CNI overlay network with kubectl apply

[caseydavenport]

I'd prefer not to use "CNI overlay network"

If anything, call it a "CNI network provider" or "CNI network plugin" - not all of these are overlays!

[lukemarsden]

I didn't want to mention CNI because users shouldn't have to understand what that is to use this. I can call it a "network provider" rather than "pod network" if there's a strong feeling about this, but I like "pod network" because users probably grok pods, and grok that they need a network...

I've gone with pod network everywhere in this doc for consistency and also described why you need one: "You must install a pod network add-on so that your pods can communicate with each other when they are on different hosts."

Where does this come from?

Pr-requisites should be higher up ^^

I wanted the tl;dr to be at the top. This is the start of the "full guide". Maybe I can make that clearer somehow.

@luxas 2GB came from my own experimentation. 512mb cluster nodes OOM as soon as you put workload on them. Maybe 1GB is OK but 2GB is a safe bet.

This sentence is really scary.

To me it sounds like:
We're going to install a secure k8s cluster, let's open all ports on all machines!

+1 - It should be clear which ports are required.

e.g "The machines will need to be able to reach the master on TCP X, and other nodes on TCP Y,Z"

This is tricky. By default most folks will run wide open between nodes in the cluster but they may want to lock it down. Perhaps say "full connectivity" instead of "all ports open". We can then have an advanced section that talks about how to run over wider links or in a more locked down way?

The default setup will be quite locked down and secure, so I think saying full connectivity between all machines like @jbeda suggested is best

But yes, we should have more in-depth docs somewhere also...

what about private networking? seems better to say "full connectivity is required" and something about preferring private networking.

I'm happy to write an in-depth doc that describe more precise, detailed requirements and deployment options and also explain the nitty gritty of what kubeadm is actually doing. But I'll do that in a second pass. For now I'll clarify that this only means that there should be full connectivity between the machines not between the machines and the internet. And I guess we can mention that private networks also work :)

Does anyone have a list of the TCP ports required to be open between nodes/masters? If so I will add the here. We can easily amend this doc after the release as well btw.

I've got with "A network connection with open ports between the machines (public or private network is fine)"

each other

Probably a non-objectives would fit in here also

I don't think that's necessary. What did you have in mind?

The real flow here will be

curl -sSL https://get.docker.com/ | sh
echo "deb https://packages.cloud.google.com/apt kubernetes-xenial-unstable main" > /etc/apt/sources.list.d/k8s.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
apt-get update && apt-get install -y kubeadm

@mikedanese We might want to promote kubeadm from unstable on the real release

Also point out that the kubeadm package will install kubelet, kubectl and kubernetes-cni as well, and other dependencies of the kubelet

That's not quite right (we're using Xenial's docker now), but yes, I'll update this once we have official debs.

A good start, but still there are things left to be done.
I'd like a little more in depth section somewhere that describes what kubeadm does under the hood; like when Kelsey describes Kubernetes the Hard Way

Also we should document the possible customization flags we have; service-cidr, cloud-provider, the external etcd options, etc.

A limitation right now is that kubectl logs doesn't work I think (since the node names aren't resolvable)
We should add that to a caveats section and how to work around (modifying /etc/hosts?)

Only kubeadm should be specified here, kubeadm should depend on the rest.
Also I think kubelet-plugin-cni should be named kubernetes-cni as on the debian side
@dgoodwin ^

I will rename on my end.

First docker should be installed from curl -sSL https://get.docker.com/ | sh

/kubelet and kubeadm/kubeadm/

Switch these two, kubeadm init is more important and should be described first

I wanted to put the use-case for development up first, to force people to think about whether they want that or not. I think it's likely to be more common than the security-conscious --schedule-workload=false.

securely bootstrapped

There is the cloud-provider option, we should document that if the user specifies that, it will (or should) work

LBs and PVs are possible, just not out of the box. Wouldn't it be enough to say, that the above guide does have some limitations especially within automatic and convenient solutions for LBs and PVs?

I've never seen this work, and I don't know what you have to do to your cloud provider instances to make it possible for them to provision PVs and LBs. If someone can educate me on that, though, I'd be happy to document it :)

@kubernetes/sig-cluster-lifecycle

Also should be valid for CentOS 7, and Red Hat Enterprise Linux 7.

Hmm, except on Fedora technically it should be "dnf" not "yum" for the commmand invocations. I think yum is still aliased to dnf though so they should work there. (/etc/yum.repos.d is valid in either case)

we're switching to CentOS 7 now, will update accordingly

I've verified that the following networking addons work with kubeadm clusters:

• Calico: https://github.com/projectcalico/calico-containers/tree/master/docs/cni/kubernetes/manifests/kubeadm
• Canal: https://github.com/tigera/canal/tree/master/k8s-install/kubeadm

@lukemarsden how would you like to add these? Should I just open a PR against your branch?

Cool yeah please do!

Very cool! I love how it is shaping up.

Spell out "Google Container Engine"?

The title gets a bit long, but maybe "Hello World on Google Container Engine" will fit

I'd love to see all addons move to being kubectl apply -f. It'll be much easier to install and manage. They were done the way they were done because we didn't have daemonsets at the time, etc.

I'd love to see instructions like this move to a weave owned page. It'll be easier to keep up to date as weave changes and will scale better as we document more types of add-ons.

If we have community supported add-ons (that are part of the k8s project) perhaps we should create new pages for those so that they keep this page clean?

I think a simple link here is all we should do. Having full installation instructions for each add-on just won't scale as more vendors get involved.

I agree with @jbeda.
Either link to a sub document (they should be responsible for) or link to documentation they own specifically. Keeping the install and instructions in the k8s repo simplifies community updates from within k8s, but it might be reasonable to let the vendor take care of this outside k8s. This would push issues and updates to the docs further to the vendor.

That's cool I'll move the verbose instructions over to a README on our github. Thanks!

Mention prerequisites? You'll need to have docker installed on each machine starting out.

socat is a strange enough dependency folks may be curious. If we really need it why isn't it a package dependency?

It is. All dependencies of kubelet will be installed automatically. This should definitely don't be here, as I pointed out above.

Can we structure this as a list so that we can put other options here? It'd be great to have 2 supported options out the gate.

I can include this in my PR to add Calico / Canal to the addons page (mentioned in another comment).

Based on user testing of this doc so far I think it's important to have one recommended option that is marked "easy to install", and I'd selfishly like that to be Weave Net. However it's also important to emphasize that the user has choice, so I will add an explicit call-out to Calico and Canal in the text. @caseydavenport if you can give me some words to use to describe them and the distinction between them and somewhere to link to, I'll add that here. Thanks!

@lukemarsden, I think we don't want this doc saying that any one solution is "recommended" by Kubernetes. I think @jbeda's suggestion of structuring this as a list or something similar is appropriate, so long as it's not really verbose.

I've had a go at tweaking this to be agnostic, while maintaining ease of use for the guide follower: https://github.com/caseydavenport/caseydavenport.github.io/pull/1/files

What do people think of something like this?

This assumes that folks are using weave-net. We need to avoid kubeadm looking like a weave only solution.

Ack.

Does socks shop have weave specific stuff in it? We should make sure that this doesn't look to be tied to the networking solution.

Maybe we can put it in kubernetes/kubernetes/examples for the time being?
I think the examples are on their way out from the main repo, but until the Google guys have made the kubernetes/examples repo, @lukemarsden can maintain it in the main repo

That would give a much more professional look to it.

The socks shop is not weave-specific, it's a generic microservices demo that will work on any kubernetes cluster (and lots of other things too) and lives in a vendor-independent org on github. I need to make a PR to add a NodePort version that will work with this guide and then I can replace the lukemarsden URL with one that links there instead.

We should make it clear that kubeadm is a work in progress and that these will be addressed in the fullness of time.

Absolutely. Will add text to that effect here. Thanks!

done

is kube-discovery an addon or a static pod?

It's a Deployment

this is going away as soon as kube-discovery gets merged into the apiserver, so I don't think we need to think too hard about this line of output right now

Should we assume everyone doing this non-expert guide to know the difference between master and nodes? Might be clearer to name them worker nodes. (The master is some kind of node too)

We've already debated about this a lot.
master/node is the current convention

I will explain what masters and nodes are in a short sentence here, as I have tried to do with other new concepts introduced in this doc.

Maybe explain or link to the limitations. When does data loss occur, when is HA needed.

I'll add a couple words around that here, sure.

https://cloud.weave.works/launch/k8s/weavescope.yaml?service-token=<token> should be in quotation marks.

good spot!

nit: "running Linux" is not "running Ubuntu 16.04 or Fedora 24"

perhaps go into detail (any requirements for kubeadm?ex. any machine with kernel version blah blah running docker?)

I will wordsmith this. I want to keep it simple at the top and add complexity gradually to make for a consumable doc, without oversimplifying.

I'm assuming that alternative networking would be documented here as well in the future? (ex. when Canal combines weave + flannel or 1.0.0 of Flannel)

perhaps go into detail (any requirements for kubeadm?ex. any machine with kernel version blah blah running docker?)

install what exactly? (kubectl / docker / rkt / kubeadm?)

s,machine,machine(s),g

Pr-requisites should be higher up ^^

what about private networking? seems better to say "full connectivity is required" and something about preferring private networking.

if we're running curl -sSL https://get.docker.com/ | sh to get docker, we may as well get the kubeadm binary from the github release pages...

run by applying to the master perhaps?

I think "installing" and "running" are more commonly understood verbs than "applying" for the target audience of this doc.

I don't think a dropbox link should be here...

Yes, this will be replaced by the packages that @mikedanese is going to bake as soon as kubernetes/kubernetes#33262 and kubernetes/kubernetes#32203 land. Same for RPMs, they will be hosted on packages.cloud.google.com.

I will add that this is a WIP only for kubeadm and that it can be made to work for k8s in general already.

We found a Contributor License Agreement for you (the sender of this pull request) and all commit authors, but as best as we can tell these commits were authored by someone else. If that's the case, please add them to this pull request and have them confirm that they're okay with these commits being contributed to Google. If we're mistaken and you did author these commits, just reply here to confirm.

@spiffxp note to self, this set of feature issue docs for 1.4 isn't done yet

When this is merged you must update the CHANGELOG.md with the docs link in kubernetes/kubernetes master. See this PR which removes the TODO:

https://github.com/kubernetes/kubernetes/pull/33418/files

Would be great if we could have some really generic instructions in the getting started (without packages for example for running on CoreOS):

This worked for a custom build of the latest HEAD of the implementation PR:

# Install kubectl, kubelet and kubeadm on every host
mkdir -p /opt/bin
curl -o /opt/bin/kubeadm https://s3-eu-west-1.amazonaws.com/jetstack.io-kubernetes-builds/release/v1.5.0-alpha.0.1375.a023085a5fa018/bin/linux/amd64/kubeadm
curl -o /opt/bin/hyperkube https://s3-eu-west-1.amazonaws.com/jetstack.io-kubernetes-builds/release/v1.5.0-alpha.0.1375.a023085a5fa018/bin/linux/amd64/hyperkube
chmod +x /opt/bin/kubeadm /opt/bin/hyperkube
ln -s hyperkube /opt/bin/kubectl
ln -s hyperkube /opt/bin/kubelet
curl -L https://github.com/containernetworking/cni/releases/download/v0.3.0/cni-v0.3.0.tgz | tar xvzf - -C /opt/cni/bin/
mkdir -p /opt/cni/bin

cat > /etc/systemd/system/kubelet.service <<'EOF'
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=http://kubernetes.io/docs/

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni"
Environment="KUBELET_DNS_ARGS=--cluster-dns=100.64.0.2 --cluster-domain=cluster.local"
Environment="KUBELET_EXTRA_ARGS=--v=4"
ExecStart=/opt/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_EXTRA_ARGS

Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

I also liked @errordeveloper's approach with using a docker image to bootstrap kubelet/kubeadm/cni.

Any word on these changes, @lukemarsden ? I'd like to get this in before tomorrow morning when we merge and launch.

@devin-donnelly I'll have the changes made and this PR ready to go for you before you wake up today (unless you wake up really early 😄). However @mikedanese indicated to me that he is likely to be cutting the 1.4 release on Tuesday now. Is that right? We should wait until we have the official packages referred to in this doc before we announce/release this doc.

@caseydavenport @devin-donnelly I've addressed your review feedback in my latest two commits. Thanks!

@devin-donnelly I think you can remove "Tech Review: Open Issues" now. And hopefully "Docs Review: Open Issues" as well :)

@pwittrock We fixed the changelog in https://github.com/kubernetes/kubernetes/pull/33262/files#diff-4ac32a78649ca5bdd8e0ba38b7006a1eL231 which landed this morning.

This PR is now only blocked on @mikedanese updating packages.cloud.google.com with 1.4.0 packages that include kubeadm. I will update this PR and remove the WIP tag soon as this has happened.

@lukemarsden I still have some changes I'd really much like to see

Mind adding dashboard here also? That's kind of helpful for new users

done, thanks

How about Fedora 24 and RHEL?

we haven't tested them so I'm not going to advertise that we work with them. We can do so once we have tested them.

Why is this here?
I'm running Kubernetes fine on my 1GB Raspberry Pi's, this only confuses users, as 2GB is not a minimum
I've also run Kubernetes on 512 MB droplets

Please remove

I've updated this to 1GB and mentioned that any less than that would leave very little room for a user's apps.

Why not Full connectivity between all machines in the cluster? Maybe also add (they should be able to ping each other)

Added the former.

Mention that apt-transport-https is required if it doesn't exist

added

@dgoodwin When you get the deps right, it should be just kubeadm
Deps:
kubeadm => kubelet, kubectl
kubelet => kubernetes-cni

I'll wait for @dgoodwin to ask me to change the instructions in the docs

Why is this here?

because users might reasonably want to do this. soon we should document using kubeadm for cloud provider integrations and then we can take this out.

It doesn't by default, but refer to the kubeadm reference doc if you want to do enable cloud provider integrations

we can add ref to that doc once it exists

/etc/kubernetes/admin.conf should be copied instead of kubelet.conf

fixed

I have a script for this, which we should refer to. Saying delete your machines is kind of awful 😄

systemctl stop kubelet; 
docker rm -f $(docker ps -q); mount | grep "/var/lib/kubelet/*" | awk '{print $3}' | xargs umount 1>/dev/null 2>/dev/null; 
rm -rf /var/lib/kubelet /etc/kubernetes /var/lib/etcd /etc/cni; 
ip link set cbr0 down; ip link del cbr0; 
ip link set cni0 down; ip link del cni0;
systemctl start kubelet; 

I'll put this inside <details> tag

@lukemarsden Also, please squash the commits

@luxas I've addressed your review feedback and squashed. Thanks!

The only issue left is the package installation guide, but I guess that will be cleaned up when @mikedanese have pushed the final packages

Marking what needs to change before we launch this.

These instructions need to be updated when @mikedanese's official packages land.

These instructions need to be updated when @mikedanese's official packages land.

Need to remove --use-kubernetes-version v1.4.0-beta.11 when @mikedanese's official packages land.