| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-05 | |||
| 18:49:15 | mriedem | you could run that in a venv/container, or backport that change to your internal repo | |
| 18:49:34 | pooja | mriedem: Thanks! Running it with max_rows=1000 couple of times seems to have purged the reservations table. Out of curiosity, is truncating the table safe operation during downtime though? | |
| 18:50:06 | ildikov | mriedem: hmm, I saw this error before but I couldn't tell right now whether it was during testing multi-attach or not | |
| 18:50:11 | mriedem | pooja: archive_deleted_rows takes records where deleted != 0 and moves them to the shadow table, then deletes them from the main table | |
| 18:50:21 | mriedem | shadow tables are named shadow_<table_name> | |
| 18:50:26 | mriedem | so shadow_instances, shadow_reservations, etc | |
| 18:50:37 | mriedem | ildikov: i have a feeling it's related to not using pike uca | |
| 18:50:52 | mriedem | pooja: archive deleted records should be safe about any time, but yeah certainly during downtime/maintenance | |
| 18:50:58 | mriedem | *downtime of the control plane | |
| 18:51:15 | ildikov | mriedem: in what sense do you mean? | |
| 18:51:28 | mriedem | ildikov: that's the only thing that's different about this CI run, | |
| 18:51:50 | mriedem | is that it's not using the pike uca, which is the default in devstack - so i'm guessing maybe something changed in devstack and implicitly relied on newer libvirt/qemu, | |
| 18:51:55 | mriedem | and now doesn't work if you're not using that | |
| 18:52:10 | pooja | Got it.. makes sense. It may be a good practice to run archive_deleted_rows periodically as a cronjob with low max_rows count such that there is no visible performance impact then? | |
| 18:52:10 | ildikov | mriedem: I was just wondering what's different in the setup that can lead to a permission denied error | |
| 18:52:12 | mriedem | could be that libvirtd isn't even running | |
| 18:52:36 | mriedem | pooja: i'd ask that in the #openstack-operators channel or mailing list | |
| 18:52:44 | mriedem | to see what other operators are doing | |
| 18:53:11 | pooja | mriedem: Sounds good, will do that. Thanks a lot for your help! | |
| 18:53:17 | mriedem | yw | |
| 18:53:44 | mriedem | looks like libvirtd is running http://logs.openstack.org/86/531386/2/check/tempest-full/4eacc3c/controller/logs/libvirt/ | |
| 18:56:17 | mriedem | hmm | |
| 18:56:17 | mriedem | 2018-01-05 16:50:52.125+0000: 11136: error : virSecurityDriverLookup:80 : unsupported configuration: Security driver apparmor not enabled | |
| 18:59:57 | mriedem | hmm https://review.openstack.org/#/c/451492/ | |
| 19:00:14 | SamYaple | ive definetely seen that error | |
| 19:00:15 | SamYaple | alot | |
| 19:00:44 | mriedem | which version? | |
| 19:00:51 | mriedem | or maybe we just don't care about the apparmor error | |
| 19:01:04 | ildikov | there are a couple other errors there too | |
| 19:01:26 | SamYaple | mriedem: primarily on 14.04, but that would have been with UCA libvirt 1.3.1 | |
| 19:01:53 | SamYaple | mriedem: ive seen it on 16.04 too, but only when some old 14.04 configs werent cleaned up properly | |
| 19:01:55 | mriedem | this is xenial so 1.3.1 | |
| 19:01:57 | openstackgerrit | Jay Pipes proposed openstack/nova master: add _has_provider_trees() utility function https://review.openstack.org/531474 | |
| 19:02:11 | SamYaple | i do not see it when running libvirt in containers with UCA libvirt | |
| 19:02:14 | mriedem | ii libvirt-bin 1.3.1-1ubuntu10.15 | |
| 19:02:36 | SamYaple | yep, thats the version ive seen it with, but it was 14.04 with mitaka UCA | |
| 19:03:02 | mriedem | this is just devstack with ENABLE_UBUNTU_CLOUD_ARCHIVE=False | |
| 19:03:40 | SamYaple | now that were talking about this, i dont know if that error broke anything, i just remember seeing it | |
| 19:03:44 | SamYaple | it could be a red herring | |
| 19:06:28 | openstackgerrit | melanie witt proposed openstack/nova master: Add access_url_base to console_auth_tokens table https://review.openstack.org/334614 | |
| 19:06:29 | openstackgerrit | melanie witt proposed openstack/nova master: Optionalize instance_uuid in console_auth_token_get_valid() https://review.openstack.org/481700 | |
| 19:06:29 | openstackgerrit | melanie witt proposed openstack/nova master: Add ConsoleAuthToken object https://review.openstack.org/320063 | |
| 19:06:30 | openstackgerrit | melanie witt proposed openstack/nova master: Add periodic task to clean expired console tokens https://review.openstack.org/325381 | |
| 19:06:30 | openstackgerrit | melanie witt proposed openstack/nova master: Use ConsoleAuthToken object to generate authorizations https://review.openstack.org/325414 | |
| 19:06:31 | openstackgerrit | melanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation https://review.openstack.org/333990 | |
| 19:06:31 | openstackgerrit | melanie witt proposed openstack/nova master: Add index on token_hash and instance_uuid for console_auth_tokens https://review.openstack.org/531477 | |
| 19:06:49 | clarkb | mriedem: SamYaple the issue with non UCA libvirt was segfaulting and other memory access faults | |
| 19:07:01 | clarkb | when we upgraded to UCA libvirt those went away (not sure if that is helpful) | |
| 19:07:09 | mriedem | clarkb: i'm getting a permission denied error on startup | |
| 19:07:21 | mriedem | connecting to libvirt-sock | |
| 19:07:26 | clarkb | mriedem: thats the group issue I think | |
| 19:07:36 | clarkb | (I thought we handled both possible groups but maybe that is broken?) | |
| 19:07:40 | mriedem | clarkb: right i was looking at this https://review.openstack.org/#/c/451492/15/stackrc | |
| 19:07:48 | SamYaple | clarkb: that helps me, and i think its safe to ignore those apparmor issues fwiw | |
| 19:08:15 | clarkb | SamYaple: we never root caused the memory issues because upstream basically said EOLD and upgrade | |
| 19:08:16 | melwitt | mriedem: when I separate the index add, is it supposed to get its own migration file under migrate_repo/versions/? | |
| 19:08:28 | mriedem | melwitt: yeah it would have to | |
| 19:08:32 | melwitt | oops. okay | |
| 19:08:52 | SamYaple | clarkb: upstream as in libvirt? or ubuntu? because ubuntu should be fixing that for the next 3(?) years or so | |
| 19:08:59 | SamYaple | clarkb: did you pop a bug with canonical? | |
| 19:09:11 | clarkb | SamYaple: upstream as in libvirt, ya we filed a bug against canonical/ubuntu | |
| 19:09:41 | mriedem | hmm, do we not have a worlddump anymore? | |
| 19:09:44 | mriedem | did that dump the groups? | |
| 21:15:48 | dansmith | and I dunno what to do with the randomize thing.. maybe also a flag about whether to do that or not (spread/pack) ? | |
| 21:15:57 | mriedem | dansmith: that's in that change | |
| 21:15:59 | mriedem | so that's already done | |
| 21:16:03 | dansmith | oh, | |
| 21:16:04 | leakypipes | dansmith: that's already there. | |
| 21:16:11 | dansmith | placement has a config not a query param for the randomize? | |
| 21:16:12 | mriedem | i think the idea being, if your limit is smaller, you do random, and if it's bigger, you don't | |
| 21:16:15 | leakypipes | sorry, that's why I was confused :) | |
| 21:16:15 | dansmith | gotcha | |
| 21:16:18 | mriedem | dansmith: correct | |
| 21:16:33 | mriedem | does seem like it should be a per-request on the client side thing | |
| 21:16:34 | dansmith | right I hadn't actually paid attention to that part | |
| 21:16:40 | mriedem | config-driven api behavior | |
| 21:17:03 | dansmith | the randomize? I would have kinda thought so, since cinder and nova might have different requirements for which 1000 to return | |
| 21:17:16 | mriedem | yeah | |
| 21:17:25 | mriedem | welp, i guess that's a future improvement for rocky | |
| 21:17:28 | dansmith | yeah, that's what I had been assuming, but it | |
| 21:17:33 | dansmith | is something we can do later | |
| 21:17:33 | dansmith | yeah | |
| 21:18:21 | mriedem | i wonder if we'd want to make the limit option on the nova side a multiple of the number of instances to build in a given request...or is that getting too cute? | |
| 21:18:55 | dansmith | opens a DoS though | |
| 21:19:05 | mriedem | true | |
| 21:19:24 | dansmith | I would make it static and then if people need cuteness, we can ask for safe suggestions | |
| 21:20:01 | mriedem | alright, now how to track this - could do it against the blueprint if someone is going to work on it now, else i can open a bug since it's kind of a bug on the nova side that we don't limit | |
| 21:20:11 | mriedem | i'm sure no one cares about that tracking business except me | |
| 21:20:24 | dansmith | I can cook up a patch while you decide how to appease your paperwork fetish | |
| 21:20:37 | mriedem | cool, because i'm already getting rashy | |
| 21:21:58 | edmondsw | the placement api-ref isn't linked from https://developer.openstack.org/api-guide/quick-start/ ... is that intentional or an oversight? | |
| 21:27:37 | fried_rice | mriedem leakypipes dansmith FYI using GET /allocation_candidates limiting is on the sched mtg agenda for Monday. | |
| 21:27:57 | leakypipes | cool, thanks | |
| 21:28:21 | leakypipes | edmondsw: good question... | |
| 21:28:32 | mriedem | edmondsw: likely an oversight | |
| 21:28:56 | mriedem | edmondsw: you should totally add that | |
| 21:29:05 | edmondsw | mriedem leakypipes alright, will do | |
| 21:35:25 | dansmith | I look forward to the rash of "you didn't write this conf option help text right" comments | |
| 21:36:40 | leakypipes | :w | |
| 21:40:38 | openstackgerrit | Merged openstack/nova master: VMware: add support for different firmwares https://review.openstack.org/448034 | |