| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-22 | |||
| 18:51:45 | jaypipes | sean-k-mooney: you're going to keep an aggregate membership sync'd with yet another periodic task? :( | |
| 18:52:22 | sean-k-mooney | no on servcei startup have the compute node remove itself form the aggrrate if its currently listed | |
| 18:52:50 | mriedem | sean-k-mooney: i don't think you'd want an aggregate for all non-disabled computes if you have 14K computes | |
| 18:52:51 | sean-k-mooney | then we can add it if we miss a heartbeat or someone use force down | |
| 18:53:29 | jaypipes | mriedem: I think he's saying the opposite. an agg for disabled providers. | |
| 18:53:30 | sean-k-mooney | mriedem: sure but you can do it the other way around too as you said in the review | |
| 18:53:50 | mriedem | sean-k-mooney: if we did aggregates for this yes that's what i'd do, an aggregate for only disabled computes | |
| 18:53:57 | jaypipes | oh look, a case for forbidden aggregates :) | |
| 18:54:05 | mriedem | and yeah that's what we'd use ^ | |
| 18:54:07 | sean-k-mooney | jaypipes: orgiginly in the code is said an aggrate for enabled because i fogot negitive member of was merged | |
| 18:54:23 | jaypipes | sean-k-mooney: it's the forbidden fruit. | |
| 18:55:00 | sean-k-mooney | from a db perspecitve this really would not be that expensice either way | |
| 18:55:27 | sean-k-mooney | the negitive case would be better but it will be similar | |
| 18:55:49 | jaypipes | so, I suppose, in order of preference, I would personally go with: a) finish up https://review.opendev.org/#/c/623558/ and the other cell-specific-service-listing thing mentioned in there, b) use an aggregate for disabled computes and member_of=! |
|
| 18:55:57 | sean-k-mooney | the aggrates table is just too uuid fields both of which are indexed | |
| 18:56:21 | jaypipes | actually, it's not really an either/or. can/should do https://review.opendev.org/#/c/623558/ as well as a placement request filter for !member_of a disabled aggregate. | |
| 18:57:26 | sean-k-mooney | yes they are not mutally exclucive and i think both are useful | |
| 18:58:40 | jaypipes | mriedem, sean-k-mooney: maybe adding a CONF.disabled_aggregate_uuid option that would be used to construct a placement request filter that used a member_of=!<uuid> query parameter to allocation candidates? | |
| 18:58:54 | jaypipes | or something like that.. | |
| 18:59:06 | sean-k-mooney | jaypipes: well it would be a prefilter so it would be enabled via config | |
| 18:59:26 | sean-k-mooney | or are you also suggesting a config option for the compute node too | |
| 18:59:41 | mriedem | if we use an aggregate we have to use the same uuid everywhere, | |
| 18:59:51 | mriedem | either via config or hard-coded in api/compute/scheduler | |
| 18:59:53 | sean-k-mooney | ya that is simple however | |
| 19:00:14 | sean-k-mooney | just use a uuid5 that we caluate for a constat or hardcode it | |
| 19:14:59 | efried | oo, a special aggregate UUID. It's almost like... metadata. | |
| 19:15:22 | sean-k-mooney | that is sotred in nova | |
| 19:15:31 | sean-k-mooney | not placement | |
| 19:15:57 | sean-k-mooney | clinets of placement are free to use aggreate to group rps whatever way the like | |
| 19:27:25 | sean-k-mooney | by the way before i go i noticed some strange errors in the nfv-ci job | |
| 19:27:29 | sean-k-mooney | libvirtError: Cannot access storage file '/opt/stack/data/nova/instances/b4afcea9-12d0-4f79-953c-d09dad3e5515/disk' (as uid:107, gid:107): Permission denied | |
| 19:27:50 | sean-k-mooney | i also got DiskNotFound: No disk at /opt/stack/data/nova/instances/f5f5ce5f-a616-4e5e-aa35-d3cc469f869a/disk | |
| 19:28:46 | sean-k-mooney | i have set the concurance to 1 and its reruning currently as i think that is part of the issue but i have no iday way those disk errors are showing up | |
| 19:30:37 | sean-k-mooney | if people want to take a look the logs are here http://logs.openstack.org/97/652197/11/check/tempest-nfv-multinode/e1fa793/compute/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_20_22_14_03_503486 | |
| 19:32:13 | mriedem | efried: except metadata that is unparseable by the human eye :) | |
| 19:32:46 | sean-k-mooney | other vms worked fine so im wondering if this is related to load/memory pressure. | |
| 19:41:39 | mriedem | d15ab1ed-0000-0000-0000-00000000000 :) | |
| 19:43:32 | efried | sean-k-mooney: speaking of CI... | |
| 19:44:08 | sean-k-mooney | what did i break now :) | |
| 19:44:15 | efried | I got an extremely vague question from within the blue walls today about what kinds of tests the PCI job ought to be running. | |
| 19:44:25 | efried | I hadn't a clue, but thought you might. | |
| 19:46:07 | sean-k-mooney | ideally pci passthough via alaise in the flavor + neutron sriov interface, then it would be nice to alos test cold migration/resize and possibly live migration in the neutron case | |
| 19:46:18 | efried | someone said something like the PCI job was just setting up devstack and not running any actual tests. I'm not sure how accurate that could be, or if they were looking at the wrong thing, or if at some point we disabled all the actual tests while we were working on the infra, or what. | |
| 19:46:32 | sean-k-mooney | i would just run the standared tempest test with tweeked flavors and different vnic_types | |
| 19:46:58 | sean-k-mooney | efried: it runs test but it boots like 1 vm or very few tests | |
| 19:48:02 | efried | was the PCI job just hitting SR-IOV? | |
| 19:48:17 | efried | Cause there's an SRIOV job that runs against neutron | |
| 19:48:22 | efried | not sure if those are related | |
| 19:49:17 | sean-k-mooney | the pci job was ment for testing QAT integration with flavor bassed passthough | |
| 19:49:29 | sean-k-mooney | there was a seperate one for neutron sriov testing | |
| 19:49:53 | sean-k-mooney | looking at http://52.27.155.124/pci/482200/7/ | |
| 19:50:07 | sean-k-mooney | there are not test in the test repofrt | |
| 19:50:35 | sean-k-mooney | but im not sure that means threare are not test | |
| 19:51:18 | sean-k-mooney | it looks like some test are run http://52.27.155.124/pci/482200/7/console_status.log.gz | |
| 19:52:48 | efried | I can't tell if they pass | |
| 19:54:21 | sean-k-mooney | nor can i | |
| 19:54:46 | sean-k-mooney | looking at the devstack output and the tempest config i cant see tehm creating any custom flavor either | |
| 19:55:55 | efried | how did you know where to find these logs? | |
| 19:56:11 | efried | the CI is running, just not reporting? | |
| 19:56:41 | sean-k-mooney | i clicked on an old log got a 404 and went up one directory | |
| 19:57:13 | efried | how did you know this patch had any results? Did you search for reviewedby:Intel PCI? | |
| 19:57:52 | sean-k-mooney | i looked at one of my patches that i knew it had commented on in the past | |
| 19:58:08 | sean-k-mooney | http://52.27.155.124/ you can see the rest of the logs are here for the other intel cis | |
| 19:58:28 | efried | IC | |
| 19:59:00 | sean-k-mooney | this is a ovh log server that i used ot own/payfor via intel for the ci logs | |
| 19:59:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug config opts infinite recursion https://review.opendev.org/654468 | |
| 20:00:11 | sean-k-mooney | or maybe it moved | |
| 20:00:16 | sean-k-mooney | it used to be http://intel-openstack-ci-logs.ovh | |
| 20:00:37 | efried | I think the 52. is aws | |
| 20:01:18 | sean-k-mooney | ya the pci ci used an aws log server. they moved to using our ovh one for a while i guess it moved back to aws at some point | |
| 20:01:20 | efried | yeah, that whole class A is amazon's. Not sure if that means aws, but probably | |
| 20:01:54 | sean-k-mooney | basicaly it just had to be a server that was not on the intel network | |
| 20:02:06 | efried | right, so the outside world could see it. | |
| 20:02:45 | sean-k-mooney | actully no os that you did not have to go through a 3 level security review to host an external facing service on the coperate network | |
| 20:03:10 | efried | yeah, that too | |
| 20:03:33 | sean-k-mooney | if its not on the copreate network it was way eaiser aslo the pci ci was originally in china which casued issues | |
| 20:04:07 | efried | okay, I'm going to dive over the firewall and throw some of this intel back at intel. Thanks for the help. | |
| 20:04:52 | sean-k-mooney | cool anyway my adivice would be to do something like im doing in the nfv ci test job | |
| 20:05:19 | sean-k-mooney | e.g. run standared tempest test but use a custom flavor or tweek teh vnic_type used in the tempest config | |
| 20:05:31 | sean-k-mooney | that will test 90% of the edgecases | |
| 20:05:50 | sean-k-mooney | at that point if the ci is stable it can be extended to test more neich things | |
| 20:07:13 | mriedem | dansmith: despite my typo in https://review.opendev.org/#/c/640197/ i'd like to get that in so we can move the grenade live migration job in-tree (this is under the queens backport) - because of devstack changes in queens + grenade i have to disable the ceph stuff in the grenade case, but it's either that or just drop the job altogether from queens which i'd like to avoid | |
| 20:07:48 | mriedem | the commit message attempts to explain the mess and reasoning for just punting and disabling the ceph case | |
| 20:09:26 | dansmith | okay | |
| 20:14:54 | sean-k-mooney | mriedem: speaking of ceph we have a downstream request to backport two of your patches to newton wich is eol upstrema. https://review.opendev.org/#/q/topic:bug/1635008+(status:open+OR+status:merged) i know its been quite a while but are you aware of any reaon of the top of your head why that woudl be a bad idea. | |
| 20:15:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for regression bug 1825537 https://review.opendev.org/654066 | |
| 20:15:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop source node allocations if finish_resize fails https://review.opendev.org/654067 | |
| 20:15:30 | openstack | bug 1825537 in OpenStack Compute (nova) "finish_resize failures incorrectly revert allocations" [Medium,In progress] https://launchpad.net/bugs/1825537 - Assigned to Matt Riedemann (mriedem) | |
| 20:15:34 | sean-k-mooney | it looked resonable to me at first glance but just said i woudl ask | |
| 20:15:58 | mriedem | sean-k-mooney: yes | |
| 20:16:20 | mriedem | there was a regression introduced | |
| 20:16:39 | sean-k-mooney | oh im glad i asked :) | |
| 20:16:50 | mriedem | for $5 i'll tell you the secret | |
| 20:17:54 | mriedem | https://review.opendev.org/#/q/I6fc7108817fcd9df4a342c9dabbf14ab7911d06a | |
| 20:18:00 | sean-k-mooney | https://bugs.launchpad.net/charm-nova-compute/+bug/1671422? | |
| 20:18:01 | openstack | Launchpad bug 1671422 in OpenStack nova-compute charm "charms: nova/cinder/ceph rbd integration broken on Ocata" [Critical,Fix released] - Assigned to James Page (james-page) | |
| 20:18:08 | mriedem | that might be the other one | |
| 20:18:37 | mriedem | https://review.opendev.org/#/q/Ieba216275c07ab16414065ee47e66915e9e9477d | |
| 20:19:01 | sean-k-mooney | ok so it did not handel olde connection info. so if we are to backprot it we need to backport the fixes for the regression too | |
| 20:19:35 | mriedem | sure | |