| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-05 | |||
| 21:40:25 | mtreinish | melwitt: yep, it exited with an error because it didn't match anything | |
| 21:40:44 | mriedem | jaypipes: i have a sql question | |
| 21:40:50 | melwitt | k | |
| 21:40:50 | mtreinish | that error message needs to be fixed though, it predates other non-regex selection mechanisms | |
| 21:40:56 | mriedem | i want to add an index on instance_actions_events for the action_id and event_id fields, | |
| 21:41:15 | mriedem | jaypipes: when we query those, we also include the deleted column so i'm assuming we want that in the index too right? | |
| 21:42:15 | jaypipes | mriedem: you don't *have* to do that, no... especially if the deleted column has very low cardinality | |
| 21:42:30 | jaypipes | mriedem: i.e. deleted column has very few distinct values | |
| 21:42:34 | mriedem | i've just noticed that we have deleted in most of our other indexes | |
| 21:42:36 | mtreinish | melwitt: the thing I'm confused by is the selection code in stestr I basically just copy and pasted from ostestr, so I'm surprised it's behaving differently | |
| 21:42:53 | melwitt | ah, yeah. I was wondering that | |
| 21:43:02 | jaypipes | mriedem: yeah, I know we have deleted in a lot of the indexes... | |
| 21:43:14 | mriedem | jaypipes: well, the values are 0 or positive int | |
| 21:43:36 | jaypipes | mriedem: right, but most are 0. | |
| 21:43:42 | mriedem | sure | |
| 21:43:53 | mriedem | maybe i should just run it both ways and see | |
| 21:43:58 | jaypipes | mriedem: that's like have a phone book with all dan smiths in it. | |
| 21:44:02 | mriedem | currently building a devstack with 500 of these | |
| 21:44:25 | gregwork | has anyone seen where all the nova services (conductor/scheduler/api) all start barfing with: ValueError: too many values to unpack. just rolling logs of this kind of error: https://paste.fedoraproject.org/paste/Vn7-6s6INFVJa5FqWb7huQ | |
| 21:44:31 | mriedem | there are 2 matt riedemanns in the dan smith phone book | |
| 21:44:54 | gregwork | nova-manage db sync also returns that ValueError | |
| 21:45:28 | mriedem | gregwork: looks like a config issue | |
| 21:45:28 | jaypipes | gregwork: you have bad sql_connection config option values. | |
| 21:45:32 | mriedem | yup | |
| 21:45:37 | mriedem | check your [database]/connection value | |
| 21:46:30 | gregwork | in /etc/nova/nova.conf ? | |
| 21:47:16 | mriedem | yes | |
| 21:48:53 | mtreinish | melwitt: found the issue, left a review comment | |
| 21:49:02 | melwitt | looking | |
| 21:49:07 | mtreinish | I was confused there for a second, because the flag worked fine locally for me | |
| 21:49:22 | melwitt | ah, thanks | |
| 21:50:21 | openstackgerrit | melanie witt proposed openstack/nova master: Make setenv consistent for unit, func, and api-samples https://review.openstack.org/507976 | |
| 22:02:16 | gregwork | yep that was exactly it | |
| 22:02:33 | gregwork | was trying to deploy browbeat and it modified nova.conf to add +connmon to the connection= | |
| 22:02:44 | gregwork | but the install failed however the conf's had been updated | |
| 22:03:03 | gregwork | thanks! | |
| 22:21:21 | mriedem | omg here it comes | |
| 22:21:35 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add _wait_for_action_fail_completion to InstanceHelperMixin https://review.openstack.org/509922 | |
| 22:21:35 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add live.migration.force.complete to the legacy notification whitelist https://review.openstack.org/508877 | |
| 22:21:36 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add functional migrate force_complete test https://review.openstack.org/509924 | |
| 22:21:36 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add recreate test for live migrate rollback not cleaning up dest allocs https://review.openstack.org/509923 | |
| 22:21:37 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Remove dest node allocations during live migration rollback https://review.openstack.org/509926 | |
| 22:21:37 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add functional for live migrate delete https://review.openstack.org/509925 | |
| 22:48:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Note TrustedFilter deprecation in docs https://review.openstack.org/509931 | |
| 23:10:49 | mriedem | well this is weird, i stop 1 instance and it generates 3 events http://paste.openstack.org/show/622795/ | |
| 23:13:25 | mtreinish | mriedem: all the request ids are different? | |
| 23:14:12 | mriedem | and 2 of the events don't have project ids | |
| 23:14:15 | mriedem | i bet i know what this is | |
| 23:14:22 | mriedem | sync_instance_power_state periodic | |
| 23:14:45 | mriedem | the last 2 are 10 minutes apart | |
| 23:14:51 | mriedem | which is the default on that periodic | |
| 23:16:07 | mriedem | ha, yeah, because the fake driver that i'm using doesn't change the state on the fake guest when you power it off | |
| 23:16:14 | mriedem | so the db says it's stopped but the "hypervisor" doesn't | |
| 23:16:16 | mriedem | so nova stops it | |
| 23:16:24 | mriedem | sev1 fake driver bug | |
| 23:18:57 | mtreinish | mriedem: hah, nice | |
| 23:21:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Implement power_off/power_on for the FakeDriver https://review.openstack.org/509935 | |
| 23:38:22 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add functional migrate force_complete test https://review.openstack.org/509924 | |
| 23:38:22 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add live.migration.force.complete to the legacy notification whitelist https://review.openstack.org/508877 | |
| 23:38:23 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Remove dest node allocations during live migration rollback https://review.openstack.org/509926 | |
| 23:38:23 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add functional for live migrate delete https://review.openstack.org/509925 | |
| #openstack-nova - 2017-10-06 | |||
| 00:09:35 | mriedem | mtreinish: does sourcing /opt/stack/new/devstack/lib/nova in grenade not also source the /opt/stack/new/devstack/local.conf? | |
| 00:11:11 | mtreinish | mriedem: probably not, in devstack the assumption is likely that is sourced well before lib/nova is called | |
| 00:12:16 | mriedem | alright | |
| 00:12:54 | mriedem | http://logs.openstack.org/71/508271/1/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/659a2cd/logs/new/local.conf.txt.gz | |
| 00:12:54 | mriedem | hmm, so can i even source this thing from within a post_test_hook? | |
| 00:13:05 | mriedem | i need to get the CELLSV2_SETUP value | |
| 00:15:19 | mriedem | seems like the only thing available to me is the ansible env vars from the d-g setup in http://logs.openstack.org/71/508271/1/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/659a2cd/logs/devstack-gate-setup-host.txt.gz | |
| 00:15:27 | mtreinish | mriedem: local.conf should be sourced by grenade already: https://github.com/openstack-dev/grenade/blob/03de9e0fc7f4fc50a00db5d547413e26cf0780dd/grenade.sh#L228 | |
| 00:15:29 | mriedem | so i can get like GRENADE_OLD_BRANCH | |
| 00:16:56 | mriedem | yeah but this is a script running in the post_test_hook | |
| 00:17:01 | mriedem | would that carry over? | |
| 00:18:31 | mriedem | http://logs.openstack.org/71/508271/1/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/659a2cd/console.html#_2017-10-04_14_36_43_557726 | |
| 00:18:35 | mtreinish | ah, ok. Yeah it won't be sourced there | |
| 00:18:45 | melwitt | mtreinish: still failing after installing python3-rados, even though it shows up in py3 pip freeze on the job. was not expecting that. and when I try locally I can import rados in py3 http://logs.openstack.org/63/509663/3/experimental/gate-tempest-dsvm-py35-full-devstack-plugin-ceph-ubuntu-xenial-nv/2361fe2/logs/screen-g-api.txt.gz?level=ERROR | |
| 00:18:51 | mriedem | ok, so i need to do https://github.com/openstack-dev/grenade/blob/03de9e0fc7f4fc50a00db5d547413e26cf0780dd/grenade.sh#L228 from within the script | |
| 00:19:23 | melwitt | back to the drawing board | |
| 00:20:34 | mtreinish | melwitt: what about rbd on py3. Looking at the glance_store code it if either rados or rbd isn't present it'll set both to None | |
| 00:20:49 | mtreinish | melwitt: https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/rbd.py#L37-L42 | |
| 00:21:08 | melwitt | mtreinish: hm, good point. lemme go down that road | |
| 00:21:35 | melwitt | yeah you're right | |
| 00:21:38 | melwitt | thanks | |
| 00:21:51 | melwitt | need to get py3 rbd separately | |
| 00:21:56 | mtreinish | sure, np | |
| 00:22:10 | mtreinish | mriedem: probably | |
| 00:22:30 | mtreinish | mriedem: or something like that to source the vars you need from the conf file | |
| 00:29:32 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix live migration grenade ceph setup https://review.openstack.org/508271 | |
| 00:29:33 | mriedem | mtreinish: check out this crazy shit ^ | |
| 02:41:19 | vivsoni | i am trying to attach FC, then attach ISCSI volume to nova instance.. while attaching it create a proper vlun entries in /dev/disk/by-pth/ | |
| 02:41:55 | vivsoni | but after detaching the iscsi volume some of the lun entries are instact in /dev/disk/by-path directory | |
| 02:44:11 | vivsoni | it would be great if someone can point me to code, from where the lun entries are created in /dev/disk/by-path | |
| 02:45:28 | vivsoni | i suspect some 'iscsiadm' or similar 'rescan' command is triggered and due to which the lun entries are create in /dev/disk/by-path... please correct me if i am worng | |
| 02:45:33 | vivsoni | thanks !!! | |
| 03:37:05 | mriedem | jaypipes: dansmith: i did it! http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-n-sch.txt.gz#_Oct_04_18_32_08_753794 | |
| 03:37:23 | mriedem | reproduced that 409 during claim resources in the scheduler, creating 1000 instances at once | |
| 03:40:09 | mriedem | retried 18 times across that 1000 | |
| 03:40:15 | mriedem | one of those poor saps just couldn't hack it | |
| 03:40:26 | mriedem | need to run with https://review.openstack.org/#/c/507705/2/nova/scheduler/client/report.py to find out which one i guess | |
| 03:41:43 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Log consumer uuid when retrying claims in the scheduler https://review.openstack.org/509961 | |
| 04:43:59 | openstackgerrit | melanie witt proposed openstack/nova master: Improve the CellDatabases test fixture and usage https://review.openstack.org/508432 | |