| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-21 | |||
| 21:16:47 | efried | I'll need to swap it back in to be more thoroughly convinced of that ^ but at least last week when I was thinking this through, I convinced myself that making these narrow/local changes was not going to help anything (given that we're not retrying on consumer alloc 409) | |
| 21:17:24 | mriedem | i agree that we don't have the context for figuring out what to do within move_allocations, | |
| 21:17:37 | mriedem | which is why i think we need a separate method that wraps move_allocations and has the necessary logic | |
| 21:17:45 | mriedem | which only gets used in the revert case | |
| 21:18:51 | mriedem | i've got about 45 min to kill so maybe i can hack something together for what i'm thinking | |
| 21:19:19 | lifeless | mriedem: constraints was parallel to solver work FWIW | |
| 21:19:33 | lifeless | mriedem: even with a solver we want stability and precise controls :) | |
| 21:20:16 | lifeless | mriedem: I did have a solver branch, but yeah, job changes at the wrong time stalled that work then someone else popped up so ... its in a branch on github somewhere but the other volunteers thing will eventuate eventually I hope | |
| 21:35:31 | mordred | lifeless: you're a solver branch | |
| 22:02:05 | mriedem | lifeless: ack | |
| 22:02:33 | mriedem | efried: ok i've got a poc which passes functional tests, i'll push that up for you and gibi to ponder before trying to cover new unit tests | |
| 22:03:02 | efried | ight | |
| 22:04:17 | mriedem | i think you're right in that there is still a race with the instance being deleted no matter how tight i make things unless we assert the existence of the instance both before and after calling move_allocations, which kind of sucks | |
| 22:15:15 | lifeless | mordred: weak sauce | |
| 22:20:48 | efried | mriedem: do you agree the proper thing would be to pull the instance allocation at the very beginning of the operation? | |
| 22:22:18 | mriedem | like before we move the instance allocations to the migration record before calling the scheduler? | |
| 22:22:37 | mriedem | or just before calling move_allocations on revert? | |
| 22:22:59 | efried | the former | |
| 22:23:43 | efried | or, does that operation return a payload? in which case keep hold of the generation from there. | |
| 22:24:37 | mriedem | does what operation return a payload? scheduling? | |
| 22:25:38 | efried | Moving allocs to the mig record. | |
| 22:26:02 | mriedem | POST /allocations returns a 204 | |
| 22:26:06 | mriedem | so no payload | |
| 22:26:08 | efried | but | |
| 22:26:14 | efried | we must have pushed a gen to do it | |
| 22:26:19 | efried | so, save that gen. | |
| 22:27:01 | mriedem | the target consumer generatoin is optional in move_allocations, which is the problem in my bug; the source consumer generation is required | |
| 22:27:04 | mriedem | but i see what you're saying | |
| 22:27:45 | efried | In English: "Make sure the instance didn't change between when we started the migration and... the bug place." | |
| 22:28:49 | mriedem | yeah idk what could go wonky with that, seems we'd only use that in an edge case, e.g. the target consumer doesn't have allocations for us to revert with a gen | |
| 22:29:20 | mriedem | for now i've just got a big comment so i can push this up | |
| 23:05:12 | melwitt | argh how do I get kibana to not show me the same change a million times for a build_status:FAILURE | |
| 23:10:40 | mriedem | you need one line from the console | |
| 23:11:42 | melwitt | ok | |
| 23:12:28 | mriedem | "ERROR: all: commands failed"? | |
| 23:12:32 | mriedem | though that's only if tempest failed | |
| 23:12:47 | mriedem | "failed: 1" would be if any play failed | |
| 23:13:47 | melwitt | aha much better. thanks | |
| 23:13:49 | mriedem | ? | |
| 23:13:49 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22failed%3A%201%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20build_name%3A%5C%22devstack-plugin-ceph-tempest%5C%22&from=7d | |
| 23:14:45 | melwitt | a lot of these look like they are the result of rebases | |
| 23:15:34 | melwitt | like, I go to the change that says it failed and there's no logs for that PS and it's not done running yet? I dunno, still looking | |
| 23:15:51 | mriedem | looking at https://d494348350733031166c-4e71828f84900af50a9a26357b84a827.ssl.cf1.rackcdn.com/689842/5/check/devstack-plugin-ceph-tempest/962455b/ | |
| 23:15:54 | mriedem | it has a few failures, | |
| 23:16:06 | mriedem | some look like http://status.openstack.org/elastic-recheck/#1763070 | |
| 23:16:23 | mriedem | and one test failed because n-api returned a 500 due to: | |
| 23:16:25 | mriedem | "NovaException: Cell 901092b8-6de2-4aad-b21a-e1c21691eb30 is not responding and hence instance info is not available." | |
| 23:16:30 | melwitt | yeah I was just looking at that one | |
| 23:16:43 | mriedem | which is likely the same thing as http://status.openstack.org/elastic-recheck/#1844929 | |
| 23:16:55 | mriedem | though ^ has only been showing up in grenade jobs (in the scheduler logs that is) | |
| 23:17:15 | melwitt | ah right | |
| 23:19:45 | melwitt | this one's an ssh timeout https://e02da289fa5cd71d2848-a802bb880ba142924be00bfc16ee185a.ssl.cf5.rackcdn.com/637647/45/check/devstack-plugin-ceph-tempest/97e930a/testr_results.html.gz | |
| 23:19:53 | melwitt | so far not seeing anything outside of known gate issues | |
| 23:21:07 | mriedem | hmm i know we disabled run_validation (ssh) in tempest api tests for that job but the ssh stuff in the scenario tests is unconditional i guess | |
| 23:21:31 | mriedem | and in that ssh fail it's due to the guest being out of space, "GROWROOT: NOCHANGE: partition 1 is size 2078687. it cannot be grown" | |
| 23:21:41 | mriedem | which we've been tracking with http://status.openstack.org/elastic-recheck/#1808010 | |
| 23:21:59 | mriedem | so maybe just get the fail dashboard fixed and we go from there | |
| 23:22:16 | melwitt | ok, yup will do | |
| 23:22:27 | mriedem | otherwise i'm seeing things like creating volumes from snapshots or images failing, which might be known races with the rbd backend in cinder | |
| 23:23:09 | melwitt | yeah, could also be races in tempest test code. I have fixed one like that in the distant past before | |
| 23:53:39 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add minor version [21] to the test_versions https://review.opendev.org/688599 | |
| #openstack-nova - 2019-10-22 | |||
| 00:51:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Check target consumer before reverting allocations https://review.opendev.org/689049 | |
| 03:58:18 | openstackgerrit | zhufl proposed openstack/nova master: [Trivial]Add missing ws between words https://review.opendev.org/689921 | |
| 07:52:31 | bauzas | good morning Nova | |
| 07:56:28 | gibi | bauzas: good morning | |
| 07:57:33 | gibi | bauzas: which provider did you use? | |
| 07:58:04 | bauzas | gibi: just paid some "MySIM" vendor with amazon marketplace | |
| 07:58:14 | bauzas | I'll see how it goes | |
| 07:58:22 | bauzas | 20EUR for 4GB and 10 days | |
| 07:58:52 | bauzas | gibi: https://www.amazon.fr/MySim-pr%C3%A9pay%C3%A9e-donn%C3%A9es-Roaming-Mobile/dp/B07LC32XMB/ref=cm_cr_arp_d_product_top?ie=UTF8 (sorry, French) | |
| 07:59:16 | gibi | bauzas: thanks | |
| 08:01:17 | bauzas | my phone is dual-SIM so I don't bother with having text and phone calls | |
| 08:02:38 | gibi | bauzas: sounds like a good plan, my worldsim offers 2 GB for $128 in china | |
| 08:03:13 | bauzas | I could pay some plan to my usual provider | |
| 08:03:21 | bauzas | but it's waaay more expensive | |
| 08:03:26 | gibi | yeah | |
| 08:04:41 | bauzas | that said, if I was subscribing to another French provider called 'Free', then I'd get 25GB of data roaming by month, including China | |
| 08:04:52 | bauzas | but that's 3G (even not 3G+) | |
| 08:05:04 | bauzas | I mean, HSDPA | |
| 08:07:49 | gibi | I think my current local provider offest wordwide (inc China) data 10EUR for 2GB for 3 days which seems usable | |
| 08:09:43 | bauzas | that's indeed reasonable, tho a bit expensive | |
| 08:10:27 | bauzas | gibi: the fact is, my hotel is far from the conference venue plus I'll have the week-end there | |
| 08:10:33 | alex_xu | $128 is crazy | |
| 08:10:42 | bauzas | gibi: I can't just rely on Wifi connections | |
| 08:11:03 | gibi | alex_xu: indeed it is | |
| 08:11:26 | gibi | bauzas: yeah, I also have the weekend before the summit in china | |
| 08:11:39 | gibi | arriving saturday morning | |
| 08:12:27 | bauzas | gibi: heh, same here :) | |
| 08:12:49 | bauzas | gibi: https://www.kayak.fr/trips/!B8vXZqcWMAbgwkrd | |
| 08:12:57 | gibi | that saturday will be crazy to survive after the long flight | |
| 08:15:08 | bauzas | I plan to sleep on plane | |
| 08:15:29 | bauzas | I did it for going to Sydney as I made one stop in HK for the day | |
| 08:15:34 | bauzas | it worked quite good | |
| 08:15:40 | gibi | bauzas: I arrive with OS75 to PVG, landing at 06:50 in China | |
| 08:15:57 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Revert "vif: Resolve a TODO and update another" https://review.opendev.org/675776 | |
| 08:16:00 | bauzas | didn't know OpenStack was running as airline :) | |
| 08:16:18 | gibi | Austrian Airlines | |
| 08:16:20 | gibi | :) | |
| 08:16:28 | gibi | but the ticker fits well :) | |
| 08:16:31 | bauzas | :p | |