| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-12 | |||
| 17:42:31 | sean-k-mooney | tasks: | |
| 17:42:33 | sean-k-mooney | - name: allocate hugepages | |
| 17:42:35 | sean-k-mooney | shell: echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages | |
| 17:42:37 | sean-k-mooney | become: yes | |
| 17:43:02 | sean-k-mooney | that will alocate 1G on the contoler and 4 on the compute of 2MB hugepages | |
| 17:43:12 | sean-k-mooney | that should be enough | |
| 17:43:55 | artom | Ack | |
| 17:55:47 | artom | sean-k-mooney, I think the problem with using https://zuul-ci.org/docs/zuul-jobs/general-roles.html#role-copy-build-sshkey is that it's 1-user only | |
| 17:55:57 | artom | IOW, 'tempest' can ssh to 'tempest' | |
| 17:56:13 | artom | We need 'tempest' to SSH to 'stack' or some other account that has limitless sudo powers | |
| 17:59:25 | sean-k-mooney | artom: the same key is already copied to root and stack | |
| 17:59:32 | sean-k-mooney | root is done in the pre job | |
| 17:59:43 | sean-k-mooney | stack in the muntinode playbook | |
| 18:00:17 | sean-k-mooney | so if we use it for tempest all 3 users will have the same private and public key | |
| 18:00:59 | sean-k-mooney | you wouldnt want to do that in production but it should be fine for ci | |
| 18:01:30 | artom | *facepalm* | |
| 18:01:31 | artom | I see | |
| 18:01:52 | artom | The devstack orchestrate-devstack role will put the pubkey in stack's authorized_keys for us | |
| 18:02:03 | artom | And we can put the private key in tempest's .ssh | |
| 18:02:13 | artom | In theory, anyways | |
| 18:05:53 | artom | ... do we have access to another project's roles? Not sure... | |
| 18:06:10 | donnyd | artom: the expanded flavors have more memory | |
| 18:11:45 | sean-k-mooney | artom: am we can get acess to them yes there are two ways to do i but your job inherits form the tempest job yes | |
| 18:11:52 | sean-k-mooney | if so then it will do it already | |
| 18:12:18 | sean-k-mooney | you you can just invoke the copy-build-key role twice once for stack and once for tempest | |
| 18:28:03 | artom | https://review.opendev.org/#/c/691062/57 let's see how badly stuff explodes | |
| 18:41:17 | sean-k-mooney | artom: you do not need the ssh keys in that anymore | |
| 18:41:36 | sean-k-mooney | but i think it should be ok | |
| 18:41:59 | sean-k-mooney | its runing in a sperate pool so it should be pretty fast | |
| 18:42:15 | artom | sean-k-mooney, something borked, it's retrying now | |
| 18:42:23 | artom | Heh, oh yeah, I should get rid of the keys | |
| 18:42:29 | artom | I'll wait for it to work first | |
| 18:43:00 | sean-k-mooney | the console for the job is here http://zuul.openstack.org/stream/11de9dc2d4fd4f7d931eedeea276af3e?logfile=console.log | |
| 18:43:23 | sean-k-mooney | looks like tis doing the normal multinode setup | |
| 18:44:30 | artom | That's the second attempt | |
| 18:44:40 | sean-k-mooney | yep | |
| 18:44:59 | sean-k-mooney | ill leave it open on one of my monitors and see what hapens | |
| 18:45:49 | sean-k-mooney | haha | |
| 18:45:54 | sean-k-mooney | you have a typo | |
| 18:46:36 | sean-k-mooney | artom: http://paste.openstack.org/show/785994/ | |
| 18:47:45 | sean-k-mooney | you did not include teh whitebox-compute role in the commit | |
| 18:47:52 | sean-k-mooney | or you forgot to remove that line | |
| 18:49:27 | artom | *facepalm* | |
| 18:49:37 | artom | 'git commit -a' isn't magic | |
| 18:49:53 | sean-k-mooney | it speciricly ignore untracted files | |
| 18:52:03 | artom | I know | |
| 18:52:42 | sean-k-mooney | good becaue i apparently cant spell specifcally and im not sure you woudl figure it out if you didnt :P | |
| 18:55:33 | sean-k-mooney | i need to try enabling spell check for irc again but the last time i tried that it was kind of depressing | |
| 18:57:29 | artom | *shrug* I forgive you :) | |
| 18:57:48 | artom | You have other uses besides spelling ;) | |
| 18:58:31 | sean-k-mooney | well if you want a terible way to generate passwords just ask me to spell somthing | |
| 18:59:14 | artom | lulz | |
| 19:04:54 | sean-k-mooney | so looking at the job you are trying to allocate 8G of hugepages on all nodes | |
| 19:05:07 | sean-k-mooney | that will not work with the non expanded flavors | |
| 19:05:11 | sean-k-mooney | *lables | |
| 19:07:17 | artom | derp | |
| 19:46:29 | mriedem | cool, the instance_actions_events table has a detail TEXT column that is never used, and when creating the InstanceActionEvent object we pass the exc_val but it gets passed down to the DB API in a 'message' kwarg rather than 'details' so that column is never populated | |
| 19:48:08 | mriedem | so as a non-admin user you can see a particular event passed or failed but not why, e.g. NoValidHost | |
| 19:48:20 | mriedem | and only admins can see the traceback (like a fault) by default | |
| 19:49:39 | sean-k-mooney | hum ok that sound like the behavior we want. are you going to reuse the colume for something | |
| 19:50:20 | sean-k-mooney | such as storging the traceback? | |
| 19:50:32 | mriedem | the traceback is stored in the traceback column | |
| 19:50:40 | mriedem | the exc_val is not stored at all | |
| 19:50:48 | mriedem | b/c the object is using the wrong column name | |
| 19:51:07 | sean-k-mooney | sure but i was wondering if you planned to chagne that | |
| 19:52:03 | mriedem | i think it would be useful to at least expose to the non-admin owner of the server the exception type, like we do for a fault message | |
| 19:53:37 | sean-k-mooney | ya maybe. as long as its just the type and not the body of the excption its proably safe. although im not sure all operator would be ok with that | |
| 19:53:55 | sean-k-mooney | e.g. it might affect there sla depening on what the falut was | |
| 19:54:13 | mriedem | if the instance goes to ERROR status we're already showing the exception type in the fault message | |
| 19:54:25 | mriedem | if not the formatted message | |
| 19:54:30 | sean-k-mooney | ya that is ture | |
| 19:54:47 | sean-k-mooney | i normally am logged in as root but we get the message in that case at least | |
| 19:54:56 | sean-k-mooney | * admin | |
| 19:55:05 | sean-k-mooney | so im not sure what a non admin sees | |
| 20:10:55 | sean-k-mooney | artom: :) looks like it worked https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_3ae/691062/59/check/whitebox-multinode-devstack/3ae547c/testr_results.html.gz | |
| 20:11:29 | sean-k-mooney | we still need to adress the skips | |
| 20:12:14 | sean-k-mooney | but the ssh keys worked correctly and it looks liek the hugepages allocation did not cause memory issues | |
| 20:23:15 | mriedem | dansmith: your suggestions here to use the tuples for comparing times doesn't work in all cases https://review.opendev.org/#/c/636224/50/nova/compute/api.py@4956 | |
| 20:23:51 | mriedem | and i don't really want to bake a bunch of conditional logic into unreadable tuples | |
| 20:25:22 | dansmith | aight, well, do what you want | |
| 20:25:32 | dansmith | inlining all that in a closure seems bad, | |
| 20:25:46 | dansmith | especially for testability, which has clearly manifested itself already | |
| 20:26:09 | dansmith | that could be a | |
| 20:26:16 | dansmith | def newer_than(obj) | |
| 20:26:42 | dansmith | method on NovaObject and be usable for other things, as well as easier to test | |
| 20:27:11 | dansmith | is there not some way we could do this without times? | |
| 20:27:22 | dansmith | because that's kinda smelly in general, | |
| 20:27:33 | sean-k-mooney | i was going to suggest a before or after method or something but ya the tuple way would be nice if it was not for the value error | |
| 20:27:37 | dansmith | assuming they're all time-synced, you don't have any delays, delayed transactions, etc | |
| 20:27:45 | sean-k-mooney | *type error | |
| 20:29:01 | sean-k-mooney | oh this is in get_newer_job already | |
| 20:29:31 | sean-k-mooney | you could define a lessthan operator on the migration object | |
| 20:29:45 | sean-k-mooney | although i guess that would have the same proably with None | |
| 20:29:57 | dansmith | no, it wouldn't | |
| 20:30:08 | dansmith | but also, less-than can mean lots of things other than time | |
| 20:30:20 | sean-k-mooney | for a migration object | |
| 20:30:26 | sean-k-mooney | and ya i could | |
| 20:30:42 | mriedem | we could just say f it and do like get_all and filter out by uuid after we've already processed one | |
| 20:31:13 | mriedem | this https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2883 | |
| 20:31:49 | dansmith | mriedem: yeah | |