| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-12 | |||
| 17:35:58 | artom | And NUMA nodes | |
| 17:36:02 | sean-k-mooney | but have hw:numa_nodes=2 | |
| 17:36:25 | sean-k-mooney | and hw:cpu_socket=2 and hw:cpu_thread=2 | |
| 17:36:43 | artom | So the non-expanded one is fine | |
| 17:37:07 | sean-k-mooney | we shoudl try to use the non expanded ones if we can to save space but we might need the expanded one on the contoler | |
| 17:37:57 | artom | Ohh | |
| 17:37:58 | sean-k-mooney | ya i didnt want to have to super optimise the job to get it to fit intially which is one of the reason we started with the expanded lables but it should not be needed longterm | |
| 17:38:37 | artom | Let's try with non-expanded I guess first | |
| 17:38:43 | artom | And if we hit oom or something we can -expand :D | |
| 17:39:06 | sean-k-mooney | yep i think i was trying to allocte a low amout of hugepage in my job | |
| 17:39:15 | sean-k-mooney | well in the orginal version | |
| 17:40:00 | sean-k-mooney | i think we are generaly around the 5-6 G mark on used memroy on the contoler so you dont want to exceen more then about 1G of hugepages total | |
| 17:40:37 | artom | Ohh, you actually set hugepages in Ansible | |
| 17:40:51 | sean-k-mooney | yes here https://review.opendev.org/#/c/679656/13/playbooks/nfv/nfv.yaml | |
| 17:40:56 | artom | So I should probably do that instead of https://review.opendev.org/#/c/693690/ | |
| 17:41:01 | sean-k-mooney | but its too much currently for the contoler | |
| 17:41:07 | artom | Although that skip could still be useful in other envs | |
| 17:42:17 | sean-k-mooney | ya i would basically do this | |
| 17:42:19 | sean-k-mooney | hosts: controller | |
| 17:42:21 | sean-k-mooney | tasks: | |
| 17:42:23 | sean-k-mooney | - name: allocate hugepages | |
| 17:42:25 | sean-k-mooney | shell: echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages | |
| 17:42:27 | sean-k-mooney | become: yes | |
| 17:42:29 | sean-k-mooney | - hosts: subnode | |
| 17:42:31 | sean-k-mooney | tasks: | |
| 17:42:33 | sean-k-mooney | - name: allocate hugepages | |
| 17:42:35 | sean-k-mooney | shell: echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages | |
| 17:42:37 | sean-k-mooney | become: yes | |
| 17:43:02 | sean-k-mooney | that will alocate 1G on the contoler and 4 on the compute of 2MB hugepages | |
| 17:43:12 | sean-k-mooney | that should be enough | |
| 17:43:55 | artom | Ack | |
| 17:55:47 | artom | sean-k-mooney, I think the problem with using https://zuul-ci.org/docs/zuul-jobs/general-roles.html#role-copy-build-sshkey is that it's 1-user only | |
| 17:55:57 | artom | IOW, 'tempest' can ssh to 'tempest' | |
| 17:56:13 | artom | We need 'tempest' to SSH to 'stack' or some other account that has limitless sudo powers | |
| 17:59:25 | sean-k-mooney | artom: the same key is already copied to root and stack | |
| 17:59:32 | sean-k-mooney | root is done in the pre job | |
| 17:59:43 | sean-k-mooney | stack in the muntinode playbook | |
| 18:00:17 | sean-k-mooney | so if we use it for tempest all 3 users will have the same private and public key | |
| 18:00:59 | sean-k-mooney | you wouldnt want to do that in production but it should be fine for ci | |
| 18:01:30 | artom | *facepalm* | |
| 18:01:31 | artom | I see | |
| 18:01:52 | artom | The devstack orchestrate-devstack role will put the pubkey in stack's authorized_keys for us | |
| 18:02:03 | artom | And we can put the private key in tempest's .ssh | |
| 18:02:13 | artom | In theory, anyways | |
| 18:05:53 | artom | ... do we have access to another project's roles? Not sure... | |
| 18:06:10 | donnyd | artom: the expanded flavors have more memory | |
| 18:11:45 | sean-k-mooney | artom: am we can get acess to them yes there are two ways to do i but your job inherits form the tempest job yes | |
| 18:11:52 | sean-k-mooney | if so then it will do it already | |
| 18:12:18 | sean-k-mooney | you you can just invoke the copy-build-key role twice once for stack and once for tempest | |
| 18:28:03 | artom | https://review.opendev.org/#/c/691062/57 let's see how badly stuff explodes | |
| 18:41:17 | sean-k-mooney | artom: you do not need the ssh keys in that anymore | |
| 18:41:36 | sean-k-mooney | but i think it should be ok | |
| 18:41:59 | sean-k-mooney | its runing in a sperate pool so it should be pretty fast | |
| 18:42:15 | artom | sean-k-mooney, something borked, it's retrying now | |
| 18:42:23 | artom | Heh, oh yeah, I should get rid of the keys | |
| 18:42:29 | artom | I'll wait for it to work first | |
| 18:43:00 | sean-k-mooney | the console for the job is here http://zuul.openstack.org/stream/11de9dc2d4fd4f7d931eedeea276af3e?logfile=console.log | |
| 18:43:23 | sean-k-mooney | looks like tis doing the normal multinode setup | |
| 18:44:30 | artom | That's the second attempt | |
| 18:44:40 | sean-k-mooney | yep | |
| 18:44:59 | sean-k-mooney | ill leave it open on one of my monitors and see what hapens | |
| 18:45:49 | sean-k-mooney | haha | |
| 18:45:54 | sean-k-mooney | you have a typo | |
| 18:46:36 | sean-k-mooney | artom: http://paste.openstack.org/show/785994/ | |
| 18:47:45 | sean-k-mooney | you did not include teh whitebox-compute role in the commit | |
| 18:47:52 | sean-k-mooney | or you forgot to remove that line | |
| 18:49:27 | artom | *facepalm* | |
| 18:49:37 | artom | 'git commit -a' isn't magic | |
| 18:49:53 | sean-k-mooney | it speciricly ignore untracted files | |
| 18:52:03 | artom | I know | |
| 18:52:42 | sean-k-mooney | good becaue i apparently cant spell specifcally and im not sure you woudl figure it out if you didnt :P | |
| 18:55:33 | sean-k-mooney | i need to try enabling spell check for irc again but the last time i tried that it was kind of depressing | |
| 18:57:29 | artom | *shrug* I forgive you :) | |
| 18:57:48 | artom | You have other uses besides spelling ;) | |
| 18:58:31 | sean-k-mooney | well if you want a terible way to generate passwords just ask me to spell somthing | |
| 18:59:14 | artom | lulz | |
| 19:04:54 | sean-k-mooney | so looking at the job you are trying to allocate 8G of hugepages on all nodes | |
| 19:05:07 | sean-k-mooney | that will not work with the non expanded flavors | |
| 19:05:11 | sean-k-mooney | *lables | |
| 19:07:17 | artom | derp | |
| 19:46:29 | mriedem | cool, the instance_actions_events table has a detail TEXT column that is never used, and when creating the InstanceActionEvent object we pass the exc_val but it gets passed down to the DB API in a 'message' kwarg rather than 'details' so that column is never populated | |
| 19:48:08 | mriedem | so as a non-admin user you can see a particular event passed or failed but not why, e.g. NoValidHost | |
| 19:48:20 | mriedem | and only admins can see the traceback (like a fault) by default | |
| 19:49:39 | sean-k-mooney | hum ok that sound like the behavior we want. are you going to reuse the colume for something | |
| 19:50:20 | sean-k-mooney | such as storging the traceback? | |
| 19:50:32 | mriedem | the traceback is stored in the traceback column | |
| 19:50:40 | mriedem | the exc_val is not stored at all | |
| 19:50:48 | mriedem | b/c the object is using the wrong column name | |
| 19:51:07 | sean-k-mooney | sure but i was wondering if you planned to chagne that | |
| 19:52:03 | mriedem | i think it would be useful to at least expose to the non-admin owner of the server the exception type, like we do for a fault message | |
| 19:53:37 | sean-k-mooney | ya maybe. as long as its just the type and not the body of the excption its proably safe. although im not sure all operator would be ok with that | |
| 19:53:55 | sean-k-mooney | e.g. it might affect there sla depening on what the falut was | |
| 19:54:13 | mriedem | if the instance goes to ERROR status we're already showing the exception type in the fault message | |
| 19:54:25 | mriedem | if not the formatted message | |
| 19:54:30 | sean-k-mooney | ya that is ture | |
| 19:54:47 | sean-k-mooney | i normally am logged in as root but we get the message in that case at least | |
| 19:54:56 | sean-k-mooney | * admin | |
| 19:55:05 | sean-k-mooney | so im not sure what a non admin sees | |
| 20:10:55 | sean-k-mooney | artom: :) looks like it worked https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_3ae/691062/59/check/whitebox-multinode-devstack/3ae547c/testr_results.html.gz | |
| 20:11:29 | sean-k-mooney | we still need to adress the skips | |