Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-12
17:16:29 sean-k-mooney but yes it also highlights "The client should not repeat this request without modification."
17:16:33 dansmith none of those reasons imply retry
17:17:55 sean-k-mooney i think the only time you woudl retry is for 5xx errors
17:18:04 dansmith mriedem: so, spec because api change, and put the api change ahead of lyarwood's patch so we can actually test through it and hold merge until both are ready yeah?
17:18:31 dansmith sean-k-mooney: no, some of the 4xx errors are mentioned as retryable with no modification
17:18:36 dansmith sean-k-mooney: like 408
17:18:57 sean-k-mooney oh ok i normaly think of 503 https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/503 when i think retry
17:19:09 mriedem dansmith: if you're saying single spec, multiple changes (one for compute, one for api), then yes agree
17:19:11 sean-k-mooney oh ya request timeouts
17:19:40 dansmith mriedem: no, I mean another spec for the api change specifically just to avoid mucking up lyarwood's clean understandable spec with the spec to change the api behavior
17:20:10 efried oh sean-k-mooney, you lost your +1s from Vieri and Andriy :(
17:20:44 sean-k-mooney :( your +2 would equal both there +1s right :)
17:21:02 dansmith mriedem: I don't want to do extra work, but it seems like having lyarwood's stable rescue spec be readable in isolation without a bunch of api semantic changes would be beneficial, but maybe the api docs required for just a status change wouldn't be too big
17:21:13 efried sean-k-mooney: re+2, thanks for the update.
17:21:35 sean-k-mooney i still need to fix the typos you noted in the code but thats next on my list
17:22:06 sean-k-mooney then i need to adress dansmith's feedback on another series :)
17:23:26 mriedem dansmith: are you saying the api change would just be changing the status code from 400 to 409 rather than a microversion?
17:23:37 dansmith mriedem: no
17:24:25 mriedem then i don't think a secondary api-only spec is going to be that big of a deal
17:24:43 dansmith cripes
17:24:45 mriedem "depends on blueprint x. in x we make y work on the compute. this spec enables it in the api with a new microversion"
17:25:08 mriedem just do whatever y'all think is best
17:25:13 mriedem i haven't read the current spec
17:25:24 dansmith mriedem: I'm trying to say I think we should separate the api change into its own small spec to keep lyarwood's focused on the disk changes
17:25:42 mriedem and i just said i was ok with that right?
17:25:53 dansmith mriedem: in isolation without his change, we can make the api change to remove the hard block and say "as of microversion X you may or may not be able to rescue BFVs"
17:26:13 dansmith mriedem: it wasn't clear to me
17:26:34 dansmith because that's what I've been suggesting for ten minutes and you seemed to be disagreeing, so I was trying to explain
17:26:39 mriedem so without any compute side changes, we should be able to rescue volume-backed servers today?
17:27:17 mriedem i'm fine with whatever
17:27:27 mriedem let's just stop talking about this
17:27:29 dansmith I'm clearly the problem here, I'll just butt out
17:34:55 artom donnyd, heya, what the difference between your multi-numa and multi-numa-expanded flavors? Or where can I look to find out?
17:35:14 artom (Also, we should talk about enabling hugepages at some point, but that can wait)
17:35:22 sean-k-mooney artom: the expanded one has 16G of ram instead of 8
17:35:31 artom sean-k-mooney, that's it?
17:35:37 sean-k-mooney yep
17:35:51 sean-k-mooney otherwise they both have the same about of disk and cpu
17:35:58 artom And NUMA nodes
17:36:02 sean-k-mooney but have hw:numa_nodes=2
17:36:25 sean-k-mooney and hw:cpu_socket=2 and hw:cpu_thread=2
17:36:43 artom So the non-expanded one is fine
17:37:07 sean-k-mooney we shoudl try to use the non expanded ones if we can to save space but we might need the expanded one on the contoler
17:37:57 artom Ohh
17:37:58 sean-k-mooney ya i didnt want to have to super optimise the job to get it to fit intially which is one of the reason we started with the expanded lables but it should not be needed longterm
17:38:37 artom Let's try with non-expanded I guess first
17:38:43 artom And if we hit oom or something we can -expand :D
17:39:06 sean-k-mooney yep i think i was trying to allocte a low amout of hugepage in my job
17:39:15 sean-k-mooney well in the orginal version
17:40:00 sean-k-mooney i think we are generaly around the 5-6 G mark on used memroy on the contoler so you dont want to exceen more then about 1G of hugepages total
17:40:37 artom Ohh, you actually set hugepages in Ansible
17:40:51 sean-k-mooney yes here https://review.opendev.org/#/c/679656/13/playbooks/nfv/nfv.yaml
17:40:56 artom So I should probably do that instead of https://review.opendev.org/#/c/693690/
17:41:01 sean-k-mooney but its too much currently for the contoler
17:41:07 artom Although that skip could still be useful in other envs
17:42:17 sean-k-mooney ya i would basically do this
17:42:19 sean-k-mooney hosts: controller
17:42:21 sean-k-mooney tasks:
17:42:23 sean-k-mooney - name: allocate hugepages
17:42:25 sean-k-mooney shell: echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
17:42:27 sean-k-mooney become: yes
17:42:29 sean-k-mooney - hosts: subnode
17:42:31 sean-k-mooney tasks:
17:42:33 sean-k-mooney - name: allocate hugepages
17:42:35 sean-k-mooney shell: echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
17:42:37 sean-k-mooney become: yes
17:43:02 sean-k-mooney that will alocate 1G on the contoler and 4 on the compute of 2MB hugepages
17:43:12 sean-k-mooney that should be enough
17:43:55 artom Ack
17:55:47 artom sean-k-mooney, I think the problem with using https://zuul-ci.org/docs/zuul-jobs/general-roles.html#role-copy-build-sshkey is that it's 1-user only
17:55:57 artom IOW, 'tempest' can ssh to 'tempest'
17:56:13 artom We need 'tempest' to SSH to 'stack' or some other account that has limitless sudo powers
17:59:25 sean-k-mooney artom: the same key is already copied to root and stack
17:59:32 sean-k-mooney root is done in the pre job
17:59:43 sean-k-mooney stack in the muntinode playbook
18:00:17 sean-k-mooney so if we use it for tempest all 3 users will have the same private and public key
18:00:59 sean-k-mooney you wouldnt want to do that in production but it should be fine for ci
18:01:30 artom *facepalm*
18:01:31 artom I see
18:01:52 artom The devstack orchestrate-devstack role will put the pubkey in stack's authorized_keys for us
18:02:03 artom And we can put the private key in tempest's .ssh
18:02:13 artom In theory, anyways
18:05:53 artom ... do we have access to another project's roles? Not sure...
18:06:10 donnyd artom: the expanded flavors have more memory
18:11:45 sean-k-mooney artom: am we can get acess to them yes there are two ways to do i but your job inherits form the tempest job yes
18:11:52 sean-k-mooney if so then it will do it already
18:12:18 sean-k-mooney you you can just invoke the copy-build-key role twice once for stack and once for tempest
18:28:03 artom https://review.opendev.org/#/c/691062/57 let's see how badly stuff explodes
18:41:17 sean-k-mooney artom: you do not need the ssh keys in that anymore
18:41:36 sean-k-mooney but i think it should be ok
18:41:59 sean-k-mooney its runing in a sperate pool so it should be pretty fast
18:42:15 artom sean-k-mooney, something borked, it's retrying now
18:42:23 artom Heh, oh yeah, I should get rid of the keys
18:42:29 artom I'll wait for it to work first
18:43:00 sean-k-mooney the console for the job is here http://zuul.openstack.org/stream/11de9dc2d4fd4f7d931eedeea276af3e?logfile=console.log
18:43:23 sean-k-mooney looks like tis doing the normal multinode setup
18:44:30 artom That's the second attempt
18:44:40 sean-k-mooney yep
18:44:59 sean-k-mooney ill leave it open on one of my monitors and see what hapens
18:45:49 sean-k-mooney haha

Earlier   Later